Machine learning optimization of fluence maps for radiotherapy treatment

ABSTRACT

Systems and methods are disclosed for generating fluence maps for a radiotherapy treatment plan that uses machine learning prediction. The systems and methods include identifying image data that indicates treatment constraints for target dose areas and organs at risk areas in an anatomy of the subject, generating anatomy projection images that represent a view of the subject from respective beam angles, using a trained neural network model to generate the computer-simulated fluence map representations based on the anatomy projection images, where the fluence maps indicate a fluence distribution of the radiotherapy treatment at each of the beam angles.

TECHNICAL FIELD

Embodiments of the present disclosure pertain generally to determiningplan parameters that direct the radiation therapy performed by aradiation therapy treatment system. In particular, the presentdisclosure pertains to using machine learning technologies to determinea fluence map used in a treatment plan for a radiation therapy system.

BACKGROUND

Radiation therapy (or “radiotherapy”) can be used to treat cancers orother ailments in mammalian (e.g., human and animal) tissue. One suchradiotherapy technique is provided using a Gamma Knife, by which apatient is irradiated by a large number of low-intensity gamma rays thatconverge with high intensity and high precision at a target (e.g., atumor). Another such radiotherapy technique is provided using a linearaccelerator (linac), whereby a tumor is irradiated by high-energyparticles (e.g., electrons, protons, ions, high-energy photons, and thelike). The placement and dose of the radiation beam must be accuratelycontrolled to ensure the tumor receives the prescribed radiation, andthe placement of the beam should be such as to minimize damage to thesurrounding healthy tissue, often called the organ(s) at risk (OARs).Radiation is termed “prescribed” because a physician orders a predefinedamount of radiation to the tumor and surrounding organs similar to aprescription for medicine. Generally, ionizing radiation in the form ofa collimated beam is directed from an external radiation source toward apatient.

A specified or selectable beam energy can be used, such as fordelivering a diagnostic energy level range or a therapeutic energy levelrange. Modulation of a radiation beam can be provided by one or moreattenuators or collimators (e.g., a multi-leaf collimator (MLC)). Theintensity and shape of the radiation beam can be adjusted by collimationto avoid damaging healthy tissue (e.g., OARs) adjacent to the targetedtissue by conforming the projected beam to a profile of the targetedtissue.

The treatment planning procedure may include using a three-dimensional(3D) image of the patient to identify a target region (e.g., the tumor)and to identify critical organs near the tumor. Creation of a treatmentplan can be a time-consuming process where a planner tries to complywith various treatment objectives or constraints (e.g., dose volumehistogram (DVH), overlap volume histogram (OVH)), taking into accounttheir individual importance (e.g., weighting) in order to produce atreatment plan that is clinically acceptable. This task can be atime-consuming trial-and-error process that is complicated by thevarious OARs because as the number of OARs increases (e.g., a dozen ormore OARs for a head-and-neck treatment), so does the complexity of theprocess. OARs distant from a tumor may be easily spared from radiation,while OARs close to or overlapping a target tumor may be difficult tospare.

Traditionally, for each patient, the initial treatment plan can begenerated in an “offline” manner. The treatment plan can be developedwell before radiation therapy is delivered, such as using one or moremedical imaging techniques. Imaging information can include, forexample, images from X-rays, computed tomography (CT), nuclear magneticresonance (MR), positron emission tomography (PET), single-photonemission computed tomography (SPECT), or ultrasound. A health careprovider, such as a physician, may use 3D imaging information indicativeof the patient anatomy to identify one or more target tumors along withthe OARs near the tumor(s). The health care provider can delineate thetarget tumor that is to receive a prescribed radiation dose using amanual technique, and the health care provider can similarly delineatenearby tissue, such as organs, at risk of damage from the radiationtreatment. Alternatively or additionally, an automated tool (e.g., ABASprovided by Elekta AB, Sweden) can be used to assist in identifying ordelineating the target tumor and organs at risk. A radiation therapytreatment plan (“treatment plan”) can then be created using numericaloptimization techniques the minimize objective functions composed ofclinical and dosimetric objectives and constraints (e.g., the maximum,minimum, and fraction of dose of radiation to a fraction of the tumorvolume (“95% of target shall receive no less than 100% of prescribeddose”), and like measures for the critical organs). The optimized planis comprised of numerical parameters that specify the direction,cross-sectional shape, and intensity of each radiation beam.

The treatment plan can then be later executed by positioning the patientin the treatment machine and delivering the prescribed radiation therapydirected by the optimized plan parameters. The radiation therapytreatment plan can include dose “fractioning,” whereby a sequence ofradiation treatments is provided over a predetermined period of time(e.g., 30-45 daily fractions), with each treatment including a specifiedfraction of a total prescribed dose.

As part of the treatment planning process for radiotherapy dosing,fluence is determined and evaluated. Fluence is the density of radiationphotons or particles normal to the beam direction, whereas dose isrelated to the energy released in the material when the photons orparticles interact with the material atoms. Dose is therefore dependenton the fluence and the physics of the radiation-matter interactions.Significant planning is conducted as part of determining fluence anddosing for a particular patient and treatment plan.

Overview

In some embodiments, methods, systems and computer-readable medium areprovided for generating an optimized fluence map or set of fluence maps,used as part of one or more radiotherapy treatment plans. The methods,systems and computer-readable medium may be configured to performoperations comprising: obtaining image data corresponding to a subjectof radiotherapy treatment, the image data indicating one or more targetdose areas and one or more organs-at-risk areas in the anatomy of thesubject; generating anatomy projection images from the image data, eachanatomy projection image providing a view of the subject from arespective beam angle of the radiotherapy treatment; and using a trainedneural network model to generate estimated fluence maps based on theanatomy projection images, each of the estimated fluence maps indicatinga fluence distribution of the radiotherapy treatment at a respectivebeam angle. In these and other configurations, such a neural networkmodel may be trained with corresponding pairs of anatomy projectionimages and fluence maps, to produce the estimated fluence map(s).

In some implementations, each of the estimated fluence maps is atwo-dimensional array of beamlet weights normal to a respective beamdirection, and beam angles of the radiotherapy treatment correspond togantry angles of a radiotherapy treatment machine. Further, obtainingthe three-dimensional set of image data corresponding to a subject mayinclude obtaining and projecting image data for each gantry angle of theradiotherapy treatment machine, such that each generated anatomyprojection image represents a view of the anatomy of the subject from agiven gantry angle used to provide treatment with a given radiotherapybeam.

In some implementations, the generated estimated fluence maps are usedduring operations to calculate and optimize radiation doses in theradiotherapy treatment plan, such as for a radiotherapy treatment thatprovides a volume modulated arc therapy (VMAT) radiotherapy performed bya radiotherapy treatment machine, as multiple radiotherapy beams areshaped to achieve a modulated dose for target areas, from among multiplebeam angles, to deliver a prescribed radiation dose. For instance, aworkflow for radiotherapy planning may involve: generating a set ofestimated fluence maps using the neural network model; performingnumerical optimization with the estimated fluence maps as input to theoptimization, where the optimization incorporates radiotherapy treatmentconstraints; and producing a pareto-optimal fluence plan used in theradiotherapy treatment plan for the subject. Such a pareto-optimalfluence plan may be used to generate a set of initial control pointscorresponding to each of multiple radiotherapy beams, using arcsequencing, and then performing direct aperture optimization, togenerate a set of final control points corresponding to each of themultiple radiotherapy beams. Further, the radiotherapy treatment may beperformed using the set of final control points, as the set of finalcontrol points are used to control multi-leaf collimator (MLC) leafpositions of a radiotherapy treatment machine at a given gantry anglecorresponding to a given beam angle.

Other aspects of generating, identifying, and optimizing fluence maps,including with the use of specific neural network training arrangementsare disclosed. For example, in a testing or verification setting, afluence map produced from the neural network model, in response to aninput set of anatomy projection images, may be compared with a fluencemap produced from another source. Specific model training aspectsinvolving a generative adversarial network (GAN), conditional generativeadversarial network (cGAN), and a cycle-consistent generativeadversarial network (CycleGAN) are also disclosed.

The above overview is intended to provide an overview of subject matterof the present patent application. It is not intended to provide anexclusive or exhaustive explanation of the inventive subject matter. Thedetailed description is included to provide further information aboutthe present patent application.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsdescribe substantially similar components throughout the several views.Like numerals having different letter suffixes represent differentinstances of substantially similar components. The drawings illustrategenerally, by way of example but not by way of limitation, variousembodiments discussed in the present document.

FIG. 1 illustrates an exemplary radiotherapy system, according to someexamples.

FIGS. 2A and 2B illustrate projection views of an ellipse and anexemplary prostate target anatomy, according to some examples.

FIG. 3A illustrates an exemplary radiation therapy system that caninclude radiation therapy output configured to provide a therapy beam,according to some examples.

FIG. 3B illustrates an exemplary system including a combined radiationtherapy system and an imaging system, such as a cone beam computedtomography (CBCT) imaging system, according to some examples.

FIG. 4 illustrates a partially cut-away view of an exemplary systemincluding a combined radiation therapy system and an imaging system,such as a nuclear magnetic resonance (MR) imaging (MM) system, accordingto some examples.

FIG. 5 illustrates an exemplary Gamma Knife radiation therapy system,according to some examples.

FIGS. 6A and 6B depict the differences between an exemplary MRI imageand a corresponding CT image, respectively, according to some examples.

FIG. 7 illustrates an exemplary collimator configuration for shaping,directing, or modulating an intensity of a radiation therapy beam,according to some examples.

FIG. 8 illustrates a data flow and processes for radiotherapy plandevelopment, according to some examples.

FIG. 9 illustrates an example of fluence map optimization operations,according to some examples.

FIG. 10 illustrates an example of anatomical projection and fluence mapimages at a particular angle of a radiotherapy beam, according to someexamples.

FIG. 11 illustrates an example of anatomical projections andradiotherapy treatment constraints at multiple angles of a radiotherapytreatment, according to some examples.

FIG. 12 illustrates an example of fluence map projections at multipleangles of a radiotherapy treatment, according to some examples.

FIG. 13 illustrates pairings of anatomical projections, radiotherapytreatment constraints, and fluence map projections, at multiple anglesof a radiotherapy treatment, according to some examples.

FIG. 14 illustrates a deep learning procedure to train a model topredict fluence maps from projection image data and fluence map data,according to some examples.

FIGS. 15A and 15B respectively depict a schematic of generative anddiscriminative deep convolutional neural networks used in predictingfluence map representations, according to some examples.

FIGS. 16A and 16B respectively depict schematics of a generativeadversarial network and a cycle-consistent generative adversarialnetwork used for training a generative model for predicting fluence maprepresentations, according to some examples.

FIGS. 17 and 18 illustrate respective data flows for training and use ofa machine learning model adapted to produce simulated fluence maps,according to some examples.

FIG. 19 illustrates a method for generating a fluence map used in aradiotherapy treatment plan and generating the machine parameters todeliver the radiotherapy treatment plan, according to some examples.

FIG. 20 illustrates an exemplary block diagram of a machine on which oneor more of the methods as discussed herein can be implemented.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings which form a part hereof, and which is shown byway of illustration-specific embodiments in which the present disclosuremay be practiced.

These embodiments, which are also referred to herein as “examples,” aredescribed in sufficient detail to enable those skilled in the art topractice the disclosure, and it is to be understood that the embodimentsmay be combined, or that other embodiments may be utilized and thatstructural, logical and electrical changes may be made without departingfrom the scope of the present disclosure. The following detaileddescription is, therefore, not to be taken in a limiting sense, and thescope of the present disclosure is defined by the appended claims andtheir equivalents.

Intensity modulated radiotherapy (IMRT) and volumetric modulated arctherapy (VMAT) have become the standards of care in modern cancerradiation therapy. Creating individual patient IMRT or VMAT treatmentplans is often a trial-and-error process, weighing target dose versusOAR sparing tradeoffs, and adjusting program constraints whose effectson the plan quality metrics and the dose distribution can be verydifficult to anticipate. Indeed, the order in which the planningconstraints are adjusted can itself result in dose differences.Treatment plan quality depends on often subjective judgements by theplanner that depend on his/her experience and skill. Even the mostskilled planners still have no assurance that their plans are close tothe best possible, or whether a little or a lot of effort will result ina significantly better plan.

The present disclosure includes various techniques to improve andenhance radiotherapy treatment by generating fluence map values, as partof a model-driven fluence map optimization (FMO) process duringradiotherapy plan design. This model may comprise a trained machinelearning model, such as an artificial neural network model, which istrained to produce (predict) a computer-modeled, image-basedrepresentation of fluence map values from a given input. These fluencemap values may be subsequently used for planning and implementingradiotherapy treatment machine parameters, including the planning andoptimization of control points that control radiotherapy machineoperations to deliver radiation therapy with treatment to a patient'sdelineated anatomy.

The technical benefits of these techniques include reduced radiotherapytreatment plan creation time, improved quality in generated radiotherapytreatment plans, and the evaluation of less data or user inputs toproduce higher quality fluence map designs. Such technical benefits mayresult in many apparent medical treatment benefits, including improvedaccuracy of radiotherapy treatment, reduced exposure to unintendedradiation, and the like. The disclosed techniques may be applicable to avariety of medical treatment and diagnostic settings or radiotherapytreatment equipment and devices, including with the use of IMRT and VMATtreatment plans.

FMO is conventionally performed as a numerical computation, producing a3D dose distribution covering the target while attempting to minimizethe effect of dose on nearby OARs. As will be understood, the optimalfluence maps and the resulting 3D dose distribution produced from use ofa fluence map are often referred to as a “plan”, even though the fluence3D dose distribution must be resampled and transformed to accommodatelinac and multileaf collimator (MLC) properties to become a clinical,deliverable treatment plan. Such changes may include arc segmentationand aperture optimization operations, and other aspects oftransformation or modifications, as discussed further below withreference to FIG. 8. However, for purposes of simplicity, referencesbelow to a “plan” used below generally refer to the planned radiationdose derived from the fluence map optimization and the outcome of atrained model adapted to produce a fluence map.

Delivering the correct fluence to achieve the desired dose in tissuesinvolves a kind of forward tomography in which 2D arrays ofappropriately weighted beamlets are directed through the linac MLC frommany angles around the target. The fluence map at each beam angle, infact, is a 2D array that spans the beam' s-eye-view projection image ofthe target. Each element of the fluence map is a real number weightproportional to the intended dose in the target. VMAT radiotherapy mayhave 100 or more beams with the total numbers of beamlet weights equalto 10⁵ or more.

FMO performs an exhaustive optimization of target and OAR constraintsdependent on thousands of small beamlets aimed at the target from manydirections, and the beamlets' weights and physics parameters describingthe material fluence dispersion for each beamlet. This high-dimensionaloptimization typically starts from default initial values for theparameters, without regard to the specific patient's anatomy. Amongother techniques, the following discusses creation and training of ananatomy-dependent model of the FMO parameters so the calculation can beinitialized closer to the ideal end values for the parameters, thusreducing the time needed to produce a satisfactory fluence map.Additionally, such an anatomy-dependent model of the FMO parameters maybe adapted for verification or validation of fluence maps, andintegrated in a variety of ways for radiotherapy planning

The following paragraphs provide an overview of example radiotherapysystem implementations and treatment planning (with reference to FIGS.2A to 7), including with the use of computing systems and hardwareimplementations (with reference to FIGS. 1 and 20). The followingparagraphs also provide a discussion of considerations specific tofluence map optimization (with reference to FIGS. 8 to 9) andrepresentations of a fluence map relative to patient anatomy projections(with reference to FIGS. 10 to 13). Finally, a discussion of machinelearning techniques (with reference to FIGS. 14 to 16B) is provided formethods of training and using a machine learning model (FIGS. 17 to 19).

FIG. 1 illustrates a radiotherapy system 100 for providing radiationtherapy to a patient. The radiotherapy system 100 includes an imageprocessing device 112. The image processing device 112 may be connectedto a network 120. The network 120 may be connected to the Internet 122.The network 120 can connect the image processing device 112 with one ormore of a database 124, a hospital database 126, an oncology informationsystem (OIS) 128, a radiation therapy device 130, an image acquisitiondevice 132, a display device 134, and a user interface 136. The imageprocessing device 112 can be configured to generate radiation therapytreatment plans 142 and plan-related data to be used by the radiationtherapy device 130.

The image processing device 112 may include a memory device 116, animage processor 114, and a communication interface 118. The memorydevice 116 may store computer-executable instructions, such as anoperating system 143, radiation therapy treatment plans 142 (e.g.,original treatment plans, adapted treatment plans and the like),software programs 144 (e.g., executable implementations of artificialintelligence, deep learning neural networks, radiotherapy treatment plansoftware), and any other computer-executable instructions to be executedby the processor 114. In an example, the software programs 144 mayconvert medical images of one format (e.g., MRI) to another format(e.g., CT) by producing synthetic images, such as pseudo-CT images. Forinstance, the software programs 144 may include image processingprograms to train a predictive model for converting a medical image 146in one modality (e.g., an MRI image) into a synthetic image of adifferent modality (e.g., a pseudo CT image); alternatively, the imageprocessing programs may convert a CT image into an MRI image. In anotherexample, the software programs 144 may register the patient image (e.g.,a CT image or an MR image) with that patient's dose distribution (alsorepresented as an image) so that corresponding image voxels and dosevoxels are associated appropriately by the network. In yet anotherexample, the software programs 144 may substitute functions of thepatient images such as signed distance functions or processed versionsof the images that emphasize some aspect of the image information. Suchfunctions might emphasize edges or differences in voxel textures, or anyother structural aspect useful to neural network learning. In anotherexample, the software programs 144 may substitute functions of a dosedistribution that emphasizes some aspect of the dose information. Suchfunctions might emphasize steep gradients around the target or any otherstructural aspect useful to neural network learning. The memory device116 may store data, including medical images 146, patient data 145, andother data required to create and implement at least one radiationtherapy treatment plan 142 or data associated with at least one plan.

In yet another example, the software programs 144 may generateprojection images for a set of two-dimensional (2D) and/or 3D CT or MRimages depicting an anatomy (e.g., one or more targets and one or moreOARs) representing different views of the anatomy from one or more beamangles used to deliver radiotherapy, which may correspond to respectivegantry angles of the radiotherapy equipment. For example, the softwareprograms 144 may process the set of CT or MR images and create a stackof projection images depicting different views of the anatomy depictedin the CT or MR images from various perspectives of the radiotherapybeams, as part of generating fluence data for radiotherapy treatmentplan. For instance, one projection image may represent a view of theanatomy from 0 degrees of the gantry, a second projection image mayrepresent a view of the anatomy from 45 degrees of the gantry, and athird projection image may represent a view of the anatomy from 90degrees of the gantry, with a separate radiotherapy beam being locatedat each angle. In other examples, each projection image may represent aview of the anatomy from a particular beam angle, corresponding to theposition of the radiotherapy beam at the respective angle of the gantry.

Projection views for a simple ellipse 202 are shown schematically inFIG. 2A. In FIG. 2A, the views are oriented relative the ellipse centerand capture the shape and extent of the ellipse 202 as seen from eachangle (e.g., 0 degrees represented by view 203, 45 degrees representedby view 204, and 90 degrees represented by view 205). For example, theview of ellipse 202 when seen from a 0-degree angle relative to they-axis 206 of ellipse 202 is projected as view 203. For example, theview of ellipse 202 when seen from a 45-degree angle relative to they-axis 206 of ellipse 202 is projected as view 204. For example, theview of ellipse 202 when seen from a 90-degree angle relative to they-axis 206 of ellipse 202 is projected as view 205.

Projections of the male pelvic anatomy relative to a set of original 3DCT images 201 are shown in FIG. 2B. Selected organs at risk and targetorgans were contoured in the 3D CT image 201 and their voxels wereassigned a code value depending on the type of anatomy. Projectionimages 250 at selected angles (0 degrees, 45 degrees, and 90 degrees)about the central axis of the 3D CT image 201 can be obtained using theforward projection capability of a reconstruction process (e.g., a conebeam CT reconstruction program). Projection images can also be computedeither by directly re-creating the projection view geometry by raytracing or by Fourier reconstruction such as is used in computedtomography.

In an example, the projection image can be computed by tracing the pathof light as pixels in an image plane and simulating the effects of itsencounters with virtual objects. In some implementations, the projectionimage is generated by tracing a path from an imaginary eye (a beam's eyeview, or an MLC view) through each pixel in a virtual screen andcalculating the color of the object visible through it. Othertomographic reconstruction techniques can be utilized to generate theprojection images from the views of the anatomy depicted in the 3D CTimages 201.

For example, the set of (or collection of) 3D CT images 201 can be usedto generate one or more views of the anatomy (e.g., the bladder,prostate, seminal vesicles, rectum, first and second targets) depictedin the 3D CT images 201. The views can be from the perspective of theradiotherapy beam (e.g., as provided by the gantry of the radiotherapydevice) and, for simplicity with reference to FIG. 2B, the views aremeasured in degrees relative to the y-axis of the 3D CT images 201 andbased on a distance between the anatomy depicted in the image and theMLC. Specifically, a first view 210 represents a projection of the 3D CTimages 201 when viewed or seen from the gantry when the gantry is 0degrees relative to the y-axis and is at a given distance from theanatomy depicted in the 3D CT image 201, a second view 220 represents aprojection of the 3D CT images 201 when viewed or seen by the gantrywhen the gantry is 45 degrees relative to the y-axis and is at a givendistance from the anatomy depicted in the 3D CT image 201, and a thirdview 230 represents a projection of the 3D CT images 201 when viewed orseen by the gantry when the gantry is 90 degrees relative to the y-axis.Any other views can be provided, such as a different view at each of 360degrees around the anatomy depicted in the 3D CT images 201.

Referring back to FIG. 1, in yet another example, the software programs144 may generate graphical image representations of fluence map data(variously referred to as fluence map representations, fluence mapimages, or “fluence maps”) at various radiotherapy beam and gantryangles, using the machine learning techniques discussed herein. Inparticular, the software programs 144 may optimize information fromthese fluence map representations in machine learning-assisted aspectsof fluence map optimization. Such fluence map data is ultimately usedgenerate and refine a set of control points that control a radiotherapydevice to produce a radiotherapy beam. The control points may representthe beam intensity, gantry angle relative to the patient position, andthe leaf positions of the MLC, among other machine parameters, todeliver the dose specified by the fluence map representation.

In yet another example, the software programs 144 store a treatmentplanning software that includes a trained machine learning model, suchas a trained generative model from a generative adversarial network(GAN), conditional generative adversarial network (cGAN), or acycle-consistent generative adversarial network (CycleGAN), to generateor estimate a fluence map image representation at a given radiotherapybeam angle, based on input to the model of a projection image of theanatomy representing the view of the anatomy from the given angle, andthe treatment constraints (e.g., target doses and organs at risk) insuch anatomy. The software programs 144 may further store a function tooptimize or accept further optimization of the fluence map data, and toconvert or compute the fluence maps into machine parameters or controlpoints for a given type of radiotherapy machine (e.g., to output a beamfrom a MLC to achieve a fluence map using the MLC leaf positions). As aresult, the treatment planning software may perform a number ofcomputations to adapt the beam shape and intensity for each radiotherapybeam and gantry angle to the radiotherapy treatment constraints, and tocompute the control points for a given radiotherapy device to achievethat beam shape and intensity in the subject patient.

In addition to the memory device 116 storing the software programs 144,it is contemplated that software programs 144 may be stored on aremovable computer medium, such as a hard drive, a computer disk, aCD-ROM, a DVD, a HD, a Blu-Ray DVD, USB flash drive, a SD card, a memorystick, or any other suitable medium; and the software programs 144 whendownloaded to image processing device 112 may be executed by imageprocessor 114.

The processor 114 may be communicatively coupled to the memory device116, and the processor 114 may be configured to executecomputer-executable instructions stored thereon. The processor 114 maysend or receive medical images 146 to memory device 116. For example,the processor 114 may receive medical images 146 from the imageacquisition device 132 via the communication interface 118 and network120 to be stored in memory device 116. The processor 114 may also sendmedical images 146 stored in memory device 116 via the communicationinterface 118 to the network 120 to be either stored in database 124 orthe hospital database 126.

Further, the processor 114 may utilize software programs 144 (e.g., atreatment planning software) along with the medical images 146 andpatient data 145 to create the radiation therapy treatment plan 142.Medical images 146 may include information such as imaging dataassociated with a patient anatomical region, organ, or volume ofinterest segmentation data. Patient data 145 may include informationsuch as (1) functional organ modeling data (e.g., serial versus parallelorgans, appropriate dose response models, etc.); (2) radiation dosagedata (e.g., DVH information; or (3) other clinical information about thepatient and course of treatment (e.g., other surgeries, chemotherapy,previous radiotherapy, etc.).

In addition, the processor 114 may utilize software programs to generateintermediate data such as updated parameters to be used, for example, bya machine learning model, such as a neural network model; or generateintermediate 2D or 3D images, which may then subsequently be stored inmemory device 116. The processor 114 may subsequently then transmit theexecutable radiation therapy treatment plan 142 via the communicationinterface 118 to the network 120 to the radiation therapy device 130,where the radiation therapy plan will be used to treat a patient withradiation. In addition, the processor 114 may execute software programs144 to implement functions such as image conversion, image segmentation,deep learning, neural networks, and artificial intelligence. Forinstance, the processor 114 may execute software programs 144 that trainor contour a medical image; such software programs 144 when executed maytrain a boundary detector or utilize a shape dictionary.

The processor 114 may be a processing device, include one or moregeneral-purpose processing devices such as a microprocessor, a centralprocessing unit (CPU), a graphics processing unit (GPU), an acceleratedprocessing unit (APU), or the like. More particularly, the processor 114may be a complex instruction set computing (CISC) microprocessor, areduced instruction set computing (RISC) microprocessor, a very longinstruction Word (VLIW) microprocessor, a processor implementing otherinstruction sets, or processors implementing a combination ofinstruction sets. The processor 114 may also be implemented by one ormore special-purpose processing devices such as an application-specificintegrated circuit (ASIC), a field programmable gate array (FPGA), adigital signal processor (DSP), a System on a Chip (SoC), or the like.As would be appreciated by those skilled in the art, in some examples,the processor 114 may be a special-purpose processor, rather than ageneral-purpose processor. The processor 114 may include one or moreknown processing devices, such as a microprocessor from the Pentium™,Core™, Xeon™ or Itanium® family manufactured by Intel™, the Turion™,Athlon™, Sempron™, Opteron™, FX™, Phenom™ family manufactured by AMD™,or any of various processors manufactured by Sun Microsystems. Theprocessor 114 may also include graphical processing units such as a GPUfrom the GeForce®, Quadro®, Tesla® family manufactured by Nvidia™, GMA,Iris™ family manufactured by Intel™, or the Radeon™ family manufacturedby AMD™. The processor 114 may also include accelerated processing unitssuch as the Xeon Phi™ family manufactured by Intel™. The disclosedexamples are not limited to any type of processor(s) otherwiseconfigured to meet the computing demands of identifying, analyzing,maintaining, generating, and/or providing large amounts of data ormanipulating such data to perform the methods disclosed herein. Inaddition, the term “processor” may include more than one processor (forexample, a multi-core design or a plurality of processors each having amulti-core design). The processor 114 can execute sequences of computerprogram instructions, stored in memory device 116, to perform variousoperations, processes, methods that will be explained in greater detailbelow.

The memory device 116 can store medical images 146. In some examples,the medical images 146 may include one or more MRI images (e.g., 2D MRI,3D MRI, 2D streaming MRI, four-dimensional (4D) MRI, 4D volumetric MRI,4D cine MRI, projection images, fluence map representation images,graphical aperture images, pairing information between projection imagesand fluence map representation images, and pairing information betweenprojection images and graphical aperture images, etc.), functional MRIimages (e.g., fMRI, DCE-MRI, diffusion MRI), CT images (e.g., 2D CT,cone beam CT, 3D CT, 4D CT), ultrasound images (e.g., 2D ultrasound, 3Dultrasound, 4D ultrasound), one or more projection images representingviews of an anatomy depicted in the MRI, synthetic CT (pseudo-CT),and/or CT images at different angles of a gantry relative to a patientaxis, PET images, X-ray images, fluoroscopic images, radiotherapy portalimages, SPECT images, computer generated synthetic images (e.g.,pseudo-CT images), aperture images, graphical aperture imagerepresentations of MLC leaf positions at different gantry angles, andthe like. Further, the medical images 146 may also include medical imagedata, for instance, training images, and training images, contouredimages, and dose images. In an example, the medical images 146 may bereceived from the image acquisition device 132. Accordingly, imageacquisition device 132 may include an Mill imaging device, a CT imagingdevice, a PET imaging device, an ultrasound imaging device, afluoroscopic device, a SPECT imaging device, an integrated linac and MRIimaging device, or other medical imaging devices for obtaining themedical images of the patient. The medical images 146 may be receivedand stored in any type of data or any type of format that the imageprocessing device 112 may use to perform operations consistent with thedisclosed examples.

The memory device 116 may be a non-transitory computer-readable medium,such as a read-only memory (ROM), a phase-change random access memory(PRAM), a static random access memory (SRAM), a flash memory, a randomaccess memory (RAM), a dynamic random access memory (DRAM) such assynchronous DRAM (SDRAM), an electrically erasable programmableread-only memory (EEPROM), a static memory (e.g., flash memory, flashdisk, static random access memory) as well as other types of randomaccess memories, a cache, a register, a CD-ROM, a DVD or other opticalstorage, a cassette tape, other magnetic storage device, or any othernon-transitory medium that may be used to store information includingimage, data, or computer executable instructions (e.g., stored in anyformat) capable of being accessed by the processor 114, or any othertype of computer device. The computer program instructions can beaccessed by the processor 114, read from the ROM, or any other suitablememory location, and loaded into the RAM for execution by the processor114. For example, the memory device 116 may store one or more softwareapplications. Software applications stored in the memory device 116 mayinclude, for example, an operating system 143 for common computersystems as well as for software-controlled devices. Further, the memorydevice 116 may store an entire software application, or only a part of asoftware application, that are executable by the processor 114. Forexample, the memory device 116 may store one or more radiation therapytreatment plans 142.

The image processing device 112 can communicate with the network 120 viathe communication interface 118, which can be communicatively coupled tothe processor 114 and the memory device 116. The communication interface118 may provide communication connections between the image processingdevice 112 and radiotherapy system 100 components (e.g., permitting theexchange of data with external devices). For instance, the communicationinterface 118 may in some examples have appropriate interfacingcircuitry to connect to the user interface 136, which may be a hardwarekeyboard, a keypad, or a touch screen through which a user may inputinformation into radiotherapy system 100.

Communication interface 118 may include, for example, a network adaptor,a cable connector, a serial connector, a USB connector, a parallelconnector, a high-speed data transmission adaptor (e.g., such as fiber,USB 3.0, thunderbolt, and the like), a wireless network adaptor (e.g.,such as a Wi-Fi adaptor), a telecommunication adaptor (e.g., 3G, 4G/LTEand the like), and the like. Communication interface 118 may include oneor more digital and/or analog communication devices that permit imageprocessing device 112 to communicate with other machines and devices,such as remotely located components, via the network 120.

The network 120 may provide the functionality of a local area network(LAN), a wireless network, a cloud computing environment (e.g., softwareas a service, platform as a service, infrastructure as a service, etc.),a client-server, a wide area network (WAN), and the like. For example,network 120 may be a LAN or a WAN that may include other systems S1(138), S2 (140), and S3 (141). Systems S1, S2, and S3 may be identicalto image processing device 112 or may be different systems. In someexamples, one or more of the systems in network 120 may form adistributed computing/simulation environment that collaborativelyperforms the examples described herein. In some examples, one or moresystems S1, S2, and S3 may include a CT scanner that obtains CT images(e.g., medical images 146). In addition, network 120 may be connected toInternet 122 to communicate with servers and clients that resideremotely on the Internet.

Therefore, network 120 can allow data transmission between the imageprocessing device 112 and a number of various other systems and devices,such as the OIS 128, the radiation therapy device 130, and the imageacquisition device 132. Further, data generated by the OIS 128 and/orthe image acquisition device 132 may be stored in the memory device 116,the database 124, and/or the hospital database 126. The data may betransmitted/received via network 120, through communication interface118 in order to be accessed by the processor 114, as required.

The image processing device 112 may communicate with database 124through network 120 to send/receive a plurality of various types of datastored on database 124. For example, database 124 may include machinedata (control points) that includes information associated with aradiation therapy device 130, image acquisition device 132, or othermachines relevant to radiotherapy. Machine data information may includecontrol points, such as radiation beam size, arc placement, beam on andoff time duration, machine parameters, segments, MLC configuration,gantry speed, MRI pulse sequence, and the like. Database 124 may be astorage device and may be equipped with appropriate databaseadministration software programs. One skilled in the art wouldappreciate that database 124 may include a plurality of devices locatedeither in a central or a distributed manner.

In some examples, database 124 may include a processor-readable storagemedium (not shown). While the processor-readable storage medium in anexample may be a single medium, the term “processor-readable storagemedium” should be taken to include a single medium or multiple media(e.g., a centralized or distributed database, and/or associated cachesand servers) that store the one or more sets of computer-executableinstructions or data. The term “processor-readable storage medium” shallalso be taken to include any medium that is capable of storing orencoding a set of instructions for execution by a processor and thatcause the processor to perform any one or more of the methodologies ofthe present disclosure. The term “processor-readable storage medium”shall accordingly be taken to include, but not be limited to,solid-state memories, optical and magnetic media. For example, theprocessor-readable storage medium can be one or more volatile,non-transitory, or non-volatile tangible computer-readable media.

Image processor 114 may communicate with database 124 to read imagesinto memory device 116 or store images from memory device 116 todatabase 124. For example, the database 124 may be configured to store aplurality of images (e.g., 3D MRI, 4D MRI, 2D MRI slice images, CTimages, 2D Fluoroscopy images, X-ray images, raw data from MR scans orCT scans, Digital Imaging and Communications in Medicine (DICOM) data,projection images, graphical aperture images, etc.) that the database124 received from image acquisition device 132. Database 124 may storedata to be used by the image processor 114 when executing softwareprogram 144, or when creating radiation therapy treatment plans 142.Database 124 may store the data produced by the trained machine leaningmode, such as a neural network including the network parametersconstituting the model learned by the network and the resultingpredicted data. The image processing device 112 may receive the imagingdata, such as a medical image 146 (e.g., 2D MRI slice images, CT images,2D Fluoroscopy images, X-ray images, 3DMRI images, 4D MRI images,projection images, graphical aperture images, etc.) either from thedatabase 124, the radiation therapy device 130 (e.g., an MRI-linac),and/or the image acquisition device 132 to generate a radiation therapytreatment plan 142.

In an example, the radiotherapy system 100 can include an imageacquisition device 132 that can acquire medical images (e.g., MRIimages, 3D MRI, 2D streaming MRI, 4D volumetric MRI, CT images,cone-Beam CT, PET images, functional MRI images (e.g., Mill, DCE-MRI anddiffusion MRI), X-ray images, fluoroscopic image, ultrasound images,radiotherapy portal images, SPECT images, and the like) of the patient.Image acquisition device 132 may, for example, be an MRI imaging device,a CT imaging device, a PET imaging device, an ultrasound device, afluoroscopic device, a SPECT imaging device, or any other suitablemedical imaging device for obtaining one or more medical images of thepatient. Images acquired by the image acquisition device 132 can bestored within database 124 as either imaging data and/or test data. Byway of example, the images acquired by the image acquisition device 132can also be stored by the image processing device 112, as medical images146 in memory device 116.

In an example, the image acquisition device 132 may be integrated withthe radiation therapy device 130 as a single apparatus (e.g., anMRI-linac). Such an MRI-linac can be used, for example, to determine alocation of a target organ or a target tumor in the patient, so as todirect radiation therapy accurately according to the radiation therapytreatment plan 142 to a predetermined target.

The image acquisition device 132 can be configured to acquire one ormore images of the patient's anatomy for a region of interest (e.g., atarget organ, a target tumor, or both). Each image, typically a 2D imageor slice, can include one or more parameters (e.g., a 2D slicethickness, an orientation, and a location, etc.). In an example, theimage acquisition device 132 can acquire a 2D slice in any orientation.For example, an orientation of the 2D slice can include a sagittalorientation, a coronal orientation, or an axial orientation. Theprocessor 114 can adjust one or more parameters, such as the thicknessand/or orientation of the 2D slice, to include the target organ and/ortarget tumor. In an example, 2D slices can be determined frominformation such as a 3D MRI volume. Such 2D slices can be acquired bythe image acquisition device 132 in “real-time” while a patient isundergoing radiation therapy treatment, for example, when using theradiation therapy device 130, with “real-time” meaning acquiring thedata in at least milliseconds or less.

The image processing device 112 may generate and store radiation therapytreatment plans 142 for one or more patients. The radiation therapytreatment plans 142 may provide information about a particular radiationdose to be applied to each patient. The radiation therapy treatmentplans 142 may also include other radiotherapy information, such ascontrol points including beam angles, gantry angles, beam intensity,dose-histogram-volume information, the number of radiation beams to beused during therapy, the dose per beam, and the like.

The image processor 114 may generate the radiation therapy treatmentplan 142 by using software programs 144 such as treatment planningsoftware (such as Monaco®, manufactured by Elekta AB of Stockholm,Sweden). In order to generate the radiation therapy treatment plans 142,the image processor 114 may communicate with the image acquisitiondevice 132 (e.g., a CT device, an MRI device, a PET device, an X-raydevice, an ultrasound device, etc.) to access images of the patient andto delineate a target, such as a tumor. In some examples, thedelineation of one or more OARs, such as healthy tissue surrounding thetumor or in close proximity to the tumor may be required. Therefore,segmentation of the OAR may be performed when the OAR is close to thetarget tumor. In addition, if the target tumor is close to the OAR(e.g., prostate in near proximity to the bladder and rectum), then bysegmenting the OAR from the tumor, the radiotherapy system 100 may studythe dose distribution not only in the target but also in the OAR.

In order to delineate a target organ or a target tumor from the OAR,medical images, such as MRI images, CT images, PET images, fMRI images,X-ray images, ultrasound images, radiotherapy portal images, SPECTimages, and the like, of the patient undergoing radiotherapy may beobtained non-invasively by the image acquisition device 132 to revealthe internal structure of a body part. Based on the information from themedical images, a 3D structure of the relevant anatomical portion may beobtained. In addition, during a treatment planning process, manyparameters may be taken into consideration to achieve a balance betweenefficient treatment of the target tumor (e.g., such that the targettumor receives enough radiation dose for an effective therapy) and lowirradiation of the OAR(s) (e.g., the OAR(s) receives as low a radiationdose as possible). Other parameters that may be considered include thelocation of the target organ and the target tumor, the location of theOAR, and the movement of the target in relation to the OAR. For example,the 3D structure may be obtained by contouring the target or contouringthe OAR within each 2D layer or slice of an MRI or CT image andcombining the contour of each 2D layer or slice. The contour may begenerated manually (e.g., by a physician, dosimetrist, or health careworker using a program such as MONACO™ manufactured by Elekta AB ofStockholm, Sweden) or automatically (e.g., using a program such as theAtlas-based auto-segmentation software, ABAS™, and a successorauto-segmentation software product ADMIRE™, manufactured by Elekta AB ofStockholm, Sweden). In certain examples, the 3D structure of a targettumor or an OAR may be generated automatically by the treatment planningsoftware.

After the target tumor and the OAR(s) have been located and delineated,a dosimetrist, physician, or healthcare worker may determine a dose ofradiation to be applied to the target tumor, as well as any maximumamounts of dose that may be received by the OAR proximate to the tumor(e.g., left and right parotid, optic nerves, eyes, lens, inner ears,spinal cord, brain stem, and the like). After the radiation dose isdetermined for each anatomical structure (e.g., target tumor, OAR), aprocess known as inverse planning may be performed to determine one ormore treatment plan parameters that would achieve the desired radiationdose distribution. Examples of treatment plan parameters include volumedelineation parameters (e.g., which define target volumes, contoursensitive structures, etc.), margins around the target tumor and OARs,beam angle selection, collimator settings, and beam-on times. During theinverse-planning process, the physician may define dose constraintparameters that set bounds on how much radiation an OAR may receive(e.g., defining full dose to the tumor target and zero dose to any OAR;defining 95% of dose to the target tumor; defining that the spinal cord,brain stem, and optic structures receive ≤45 Gy, ≤55 Gy and <54 Gy,respectively). The result of inverse planning may constitute a radiationtherapy treatment plan 142 that may be stored in memory device 116 ordatabase 124. Some of these treatment parameters may be correlated. Forexample, tuning one parameter (e.g., weights for different objectives,such as increasing the dose to the target tumor) in an attempt to changethe treatment plan may affect at least one other parameter, which inturn may result in the development of a different treatment plan. Thus,the image processing device 112 can generate a tailored radiationtherapy treatment plan 142 having these parameters in order for theradiation therapy device 130 to provide radiotherapy treatment to thepatient.

In addition, the radiotherapy system 100 may include a display device134 and a user interface 136. The display device 134 may include one ormore display screens that display medical images, interface information,treatment planning parameters (e.g., projection images, graphicalaperture images, contours, dosages, beam angles, etc.) treatment plans,a target, localizing a target and/or tracking a target, or any relatedinformation to the user. The user interface 136 may be a keyboard, akeypad, a touch screen or any type of device with which a user may inputinformation to radiotherapy system 100. Alternatively, the displaydevice 134 and the user interface 136 may be integrated into a devicesuch as a tablet computer (e.g., Apple iPad®, Lenovo Thinkpad®, SamsungGalaxy®, etc.).

Furthermore, any and all components of the radiotherapy system 100 maybe implemented as a virtual machine (e.g., VMWare, Hyper-V, and thelike). For instance, a virtual machine can be software that functions ashardware. Therefore, a virtual machine can include at least one or morevirtual processors, one or more virtual memories, and one or morevirtual communication interfaces that together function as hardware. Forexample, the image processing device 112, the OIS 128, the imageacquisition device 132 could be implemented as a virtual machine. Giventhe processing power, memory, and computational capability available,the entire radiotherapy system 100 could be implemented as a virtualmachine.

FIG. 3A illustrates a radiation therapy device 302 that may include aradiation source, such as an X-ray source or a linear accelerator, acouch 316, an imaging detector 314, and a radiation therapy output 304.The radiation therapy device 302 may be configured to emit a radiationbeam 308 to provide therapy to a patient. The radiation therapy output304 can include one or more attenuators or collimators, such as an MLCas described in the illustrative example of FIG. 7, below.

Referring back to FIG. 3A, a patient can be positioned in a region 312and supported by the treatment couch 316 to receive a radiation therapydose, according to a radiation therapy treatment plan. The radiationtherapy output 304 can be mounted or attached to a gantry 306 or othermechanical support. One or more chassis motors (not shown) may rotatethe gantry 306 and the radiation therapy output 304 around couch 316when the couch 316 is inserted into the treatment area. In an example,gantry 306 may be continuously rotatable around couch 316 when the couch316 is inserted into the treatment area. In another example, gantry 306may rotate to a predetermined position when the couch 316 is insertedinto the treatment area. For example, the gantry 306 can be configuredto rotate the therapy output 304 around an axis (“A”). Both the couch316 and the radiation therapy output 304 can be independently moveableto other positions around the patient, such as moveable in transversedirection (“T”), moveable in a lateral direction (“L”), or as rotationabout one or more other axes, such as rotation about a transverse axis(indicated as “R”). A controller communicatively connected to one ormore actuators (not shown) may control the couch 316 movements orrotations in order to properly position the patient in or out of theradiation beam 308 according to a radiation therapy treatment plan. Boththe couch 316 and the gantry 306 are independently moveable from oneanother in multiple degrees of freedom, which allows the patient to bepositioned such that the radiation beam 308 can target the tumorprecisely. The MLC may be integrated and included within gantry 306 todeliver the radiation beam 308 of a certain shape.

The coordinate system (including axes A, T, and L) shown in FIG. 3A canhave an origin located at an isocenter 310. The isocenter can be definedas a location where the central axis of the radiation beam 308intersects the origin of a coordinate axis, such as to deliver aprescribed radiation dose to a location on or within a patient.Alternatively, the isocenter 310 can be defined as a location where thecentral axis of the radiation beam 308 intersects the patient forvarious rotational positions of the radiation therapy output 304 aspositioned by the gantry 306 around the axis A. As discussed herein, thegantry angle corresponds to the position of gantry 306 relative to axisA, although any other axis or combination of axes can be referenced andused to determine the gantry angle.

Gantry 306 may also have an attached imaging detector 314. The imagingdetector 314 is preferably located opposite to the radiation source, andin an example, the imaging detector 314 can be located within a field ofthe radiation beam 308.

The imaging detector 314 can be mounted on the gantry 306 (preferablyopposite the radiation therapy output 304), such as to maintainalignment with the therapy beam 308. The imaging detector 314 rotatesabout the rotational axis as the gantry 306 rotates. In an example, theimaging detector 314 can be a flat panel detector (e.g., a directdetector or a scintillator detector). In this manner, the imagingdetector 314 can be used to monitor the radiation beam 308 or theimaging detector 314 can be used for imaging the patient's anatomy, suchas portal imaging. The control circuitry of the radiation therapy device302 may be integrated within the radiotherapy system 100 or remote fromit.

In an illustrative example, one or more of the couch 316, the therapyoutput 304, or the gantry 306 can be automatically positioned, and thetherapy output 304 can establish the radiation beam 308 according to aspecified dose for a particular therapy delivery instance. A sequence oftherapy deliveries can be specified according to a radiation therapytreatment plan, such as using one or more different orientations orlocations of the gantry 306, couch 316, or therapy output 304. Thetherapy deliveries can occur sequentially, but can intersect in adesired therapy locus on or within the patient, such as at the isocenter310. A prescribed cumulative dose of radiation therapy can thereby bedelivered to the therapy locus while damage to tissue near the therapylocus can be reduced or avoided.

FIG. 3B illustrates a radiation therapy device 302 that may include acombined linac and an imaging system, such as a CT imaging system. Theradiation therapy device 302 can include an MLC (not shown). The CTimaging system can include an imaging X-ray source 318, such asproviding X-ray energy in a kiloelectron-Volt (keV) energy range. Theimaging X-ray source 318 can provide a fan-shaped and/or a conicalradiation beam 308 directed to an imaging detector 322, such as a flatpanel detector. The radiation therapy device 302 can be similar to thesystem described in relation to FIG. 3A, such as including a radiationtherapy output 304, a gantry 306, a couch 316, and another imagingdetector 314 (such as a flat panel detector). The X-ray source 318 canprovide a comparatively-lower-energy X-ray diagnostic beam, for imaging.

In the illustrative example of FIG. 3B, the radiation therapy output 304and the X-ray source 318 can be mounted on the same rotating gantry 306,rotationally separated from each other by 90 degrees. In anotherexample, two or more X-ray sources can be mounted along thecircumference of the gantry 306, such as each having its own detectorarrangement to provide multiple angles of diagnostic imagingconcurrently. Similarly, multiple radiation therapy outputs 304 can beprovided.

FIG. 4 depicts a radiation therapy system 400 that can include combininga radiation therapy device 302 and an imaging system, such as a magneticresonance (MR) imaging system (e.g., known in the art as an MR-linac)consistent with the disclosed examples. As shown, system 300 may includea couch 316, an image acquisition device 420, and a radiation deliverydevice 430. System 300 delivers radiation therapy to a patient inaccordance with a radiotherapy treatment plan. In some examples, imageacquisition device 420 may correspond to image acquisition device 132 inFIG. 1 that may acquire origin images of a first modality (e.g., MRIimage shown in FIG. 6A) or destination images of a second modality(e.g., CT image shown in FIG. 6B).

Couch 316 may support a patient (not shown) during a treatment session.In some implementations, couch 316 may move along a horizontaltranslation axis (labelled “I”), such that couch 316 can move thepatient resting on couch 316 into and/or out of system 400. Couch 316may also rotate around a central vertical axis of rotation, transverseto the translation axis. To allow such movement or rotation, couch 316may have motors (not shown) enabling the couch 316 to move in variousdirections and to rotate along various axes. A controller (not shown)may control these movements or rotations in order to properly positionthe patient according to a treatment plan.

In some examples, image acquisition device 420 may include an MRImachine used to acquire 2D or 3D MRI images of the patient before,during, and/or after a treatment session. Image acquisition device 420may include a magnet 421 for generating a primary magnetic field formagnetic resonance imaging. The magnetic field lines generated byoperation of magnet 421 may run substantially parallel to the centraltranslation axis I. Magnet 421 may include one or more coils with anaxis that runs parallel to the translation axis I. In some examples, theone or more coils in magnet 421 may be spaced such that a central window423 of magnet 421 is free of coils. In other examples, the coils inmagnet 421 may be thin enough or of a reduced density such that they aresubstantially transparent to radiation of the wavelength generated byradiotherapy device 430. Image acquisition device 420 may also includeone or more shielding coils, which may generate a magnetic field outsidemagnet 421 of approximately equal magnitude and opposite polarity inorder to cancel or reduce any magnetic field outside of magnet 421. Asdescribed below, radiation source 431 of radiation delivery device 430may be positioned in the region where the magnetic field is cancelled,at least to a first order, or reduced.

Image acquisition device 420 may also include two gradient coils 425 and426, which may generate a gradient magnetic field that is superposed onthe primary magnetic field. Coils 425 and 426 may generate a gradient inthe resultant magnetic field that allows spatial encoding of the protonsso that their position can be determined. Gradient coils 425 and 426 maybe positioned around a common central axis with the magnet 421 and maybe displaced along that central axis. The displacement may create a gap,or window, between coils 425 and 426. In examples where magnet 421 canalso include a central window 423 between coils, the two windows may bealigned with each other.

In some examples, image acquisition device 420 may be an imaging deviceother than an MRI, such as an X-ray, a CT, a CBCT, a spiral CT, a PET, aSPECT, an optical tomography, a fluorescence imaging, ultrasoundimaging, radiotherapy portal imaging device, or the like. As would berecognized by one of ordinary skill in the art, the above description ofimage acquisition device 420 concerns certain examples and is notintended to be limiting.

Radiation delivery device 430 may include the radiation source 431, suchas an X-ray source or a linac, and an MLC 432 (shown below in moredetail in FIG. 7). Radiation delivery device 430 may be mounted on achassis 435. One or more chassis motors (not shown) may rotate thechassis 435 around the couch 316 when the couch 316 is inserted into thetreatment area. In an example, the chassis 435 may be continuouslyrotatable around the couch 316, when the couch 316 is inserted into thetreatment area. Chassis 435 may also have an attached radiation detector(not shown), preferably located opposite to radiation source 431 andwith the rotational axis of the chassis 435 positioned between theradiation source 431 and the detector. Further, the device 430 mayinclude control circuitry (not shown) used to control, for example, oneor more of the couch 316, image acquisition device 420, and radiotherapydevice 430. The control circuitry of the radiation delivery device 430may be integrated within the system 400 or remote from it.

During a radiotherapy treatment session, a patient may be positioned oncouch 316. System 400 may then move couch 316 into the treatment areadefined by the magnet 421, coils 425, 426, and chassis 435. Controlcircuitry may then control radiation source 431, MLC 432, and thechassis motor(s) to deliver radiation to the patient through the windowbetween coils 425 and 426 according to a radiotherapy treatment plan.

FIG. 3A, FIG. 3B, and FIG. 4 illustrate generally examples of aradiation therapy device configured to provide radiotherapy treatment toa patient, including a configuration where a radiation therapy outputcan be rotated around a central axis (e.g., an axis “A”). Otherradiation therapy output configurations can be used. For example, aradiation therapy output can be mounted to a robotic arm or manipulatorhaving multiple degrees of freedom. In yet another example, the therapyoutput can be fixed, such as located in a region laterally separatedfrom the patient, and a platform supporting the patient can be used toalign a radiation therapy isocenter with a specified target locus withinthe patient.

FIG. 5 illustrates an example of another type of radiotherapy device 530(e.g., a Leksell Gamma Knife). As shown in FIG. 5, in a radiotherapytreatment session, a patient 502 may wear a coordinate frame 520 to keepstable the patient's body part (e.g., the head) undergoing surgery orradiotherapy. Coordinate frame 520 and a patient positioning system 522may establish a spatial coordinate system, which may be used whileimaging a patient or during radiation surgery. Radiotherapy device 530may include a protective housing 514 to enclose a plurality of radiationsources 512. Radiation sources 512 may generate a plurality of radiationbeams (e.g., beamlets) through beam channels 516. The plurality ofradiation beams may be configured to focus on an isocenter 310 fromdifferent directions. While each individual radiation beam may have arelatively low intensity, isocenter 310 may receive a relatively highlevel of radiation when multiple doses from different radiation beamsaccumulate at isocenter 310. In certain examples, isocenter 310 maycorrespond to a target under surgery or treatment, such as a tumor.

As discussed above, radiation therapy devices described by FIG. 3A, FIG.3B, and FIG. 4 include an MLC for shaping, directing, or modulating anintensity of a radiation therapy beam to the specified target locuswithin the patient. FIG. 7 illustrates an MLC 432 that includes leaves732A through 732J that can be automatically positioned to define anaperture approximating a tumor 740 cross-section or projection. Theleaves 732A through 732J permit modulation of the radiation therapybeam. The leaves 732A through 732J can be made of a material specifiedto attenuate or block the radiation beam in regions other than theaperture, in accordance with the radiation treatment plan. For example,the leaves 732A through 732J can include metallic plates, such ascomprising tungsten, with a long axis of the plates oriented parallel toa beam direction and having ends oriented orthogonally to the beamdirection (as shown in the plane of the illustration of FIG. 2A). A“state” of the MLC 432 can be adjusted adaptively during a course ofradiation therapy treatment, such as to establish a therapy beam thatbetter approximates a shape or location of the tumor 740 or other targetlocus. This is in comparison to using a static collimator configurationor as compared to using an MLC configuration determined exclusivelyusing an “offline” therapy planning technique. A radiation therapytechnique using the MLC 432 to produce a specified radiation dosedistribution to a tumor or to specific areas within a tumor can bereferred to as IMRT. The resulting beam shape that is output using theMLC 432 is represented as a graphical aperture image. Namely, a givengraphical aperture image is generated to represent how a beam looks(beam shape) and its intensity after being passed through and output byMLC 432.

IMRT techniques involve irradiating a subject patient at a small numberof fixed gantry angles; whereas VMAT techniques typically involveirradiating a subject patient from 100 or more gantry angles.Specifically, with VMAT radiotherapy devices, the patient is irradiatedcontinuously by a linac revolving around the patient with a beamcontinuously shaped by MLC producing apertures to achieve a modulatedcoverage of the target, from each angle, by a prescribed radiation dose.VMAT has become popular because it accurately irradiates targets whileminimizing dose to neighboring OARs, and VMAT treatments generally takeless time than those of IMRT.

Creating plans personalized for every patient using either IMRT or VMATis difficult. Treatment planning systems model the physics of radiationdose, but they provide little assistance to the planner to indicate howto vary treatment parameters to achieve high quality plans. Changingplan variables often produces nonintuitive results, and the treatmentplanning system is unable to tell the planner whether a little or a lotof effort will be needed to advance the current plan-in-progress to aclinically usable plan. Automated multicriteria optimization reducesplanning uncertainty by automated, exhaustive numerical optimizationssatisfying a hierarchy of target-OAR constraints, but this method istime consuming and often does not produce a deliverable plan.

Radiotherapy plan creation typically involves the application ofmultiple processes to address treatment plan considerations. FIG. 8illustrates a data flow through the three typical stages of VMAT plandevelopment: Fluence map optimization (FMO) 820, Arc sequencing 840, andDirect aperture optimization 860. As shown in FIG. 8, patient imagestructures 810, such as image data received from CT, MRI, or similarimaging modalities, are received as input for treatment planning.Through a process of fluence map optimization 820, a fluence map 830 isidentified and created. For VMAT plans, the fluence map 830 representsthe ideal target dose coverage that must be replicated by constructingsegments (MLC apertures and monitor unit weights) at a set of linacgantry angles.

Specifically, the fluence map 830 provides a model of an ideal 3D dosedistribution for a radiotherapy treatment, and is constructed duringfluence map optimization (FMO) 820. FMO is a hierarchical,multicriteria, numerical optimization that models the irradiation of thetarget with many small X-ray beamlets subject to target dose and OARconstraints. The resulting fluence maps 830 represent 2D arrays ofbeamlets' weights that map the radiation onto a beam's-eye-view of thetarget; thus, in planning a VMAT treatment, there is a fluence map foreach VMAT beam at every one of the 100 or more angle settings of thelinac gantry encircling the patient. Since fluence is the density ofrays traversing a unit surface normal to the beam direction, and dose isthe energy released in the irradiated material, the resulting 3D dosecovering the target is specified by the set of 2D fluence maps.

The 3D dose that is represented in a fluence map 830, produced from FMO820, does not include sufficient information about how a machine candeliver radiation to achieve that distribution. Therefore, an initialset of linac/MLC weighted apertures (one set per gantry angle; alsocalled a control point) must be created by iterative modelling of the 3Ddose by a succession of MLC apertures at varying gantry angles and withappropriate intensities or weights. These initial control points 850 areproduced from arc sequencing 840, with the resulting apertures andparameters of (initial control points 850) being dependent on thespecific patient's anatomy and target geometries.

Even with the generation of many control points 850, additionalrefinement of the apertures and weights is often involved, occasionallyadding or subtracting a control point. Refinement is necessary since the3D dose distribution resulting from arc sequencing 840 is degraded withrespect to the original optimal fluence map 830, and some refinement ofthe apertures invariably improves the resulting plan quality. Theprocess of optimizing the apertures of these control points is referredto as direct aperture optimization 860, with the resulting refinedapertures and weights (final control points 870) being dependent on thespecific patient's anatomy and target geometries.

In each of the operations 820, 840, 860, an achievable solutioncorresponds to the minimum value of an objective function in ahigh-dimensional space that may have many minima and requires lengthynumerical optimizations. In each case, the objective function describesa mapping or relationship between the patient's anatomic structures anda dose distribution or set of linac/MLC machine parameters. Thefollowing techniques discuss a mechanism by which the generation offluence maps 830 and the process of fluence map optimization 820 canitself be optimized from modeling. Specifically, the optimization of afluence map may occur through the generation of a fluence map using aprobabilistic model, such as with a learned model that is trained viamachine learning techniques.

Probabilistic modeling of a fluence maps, based on a model that islearned from populations of clinical plans, can provide two significantbenefits to the FMO operations discussed in FIG. 8. One benefit from useof a probabilistic model is to accelerate the search for a solution.Using a trained probabilistic model, a new patient's structures can beused to infer fluence maps (e.g., fluence maps 830) that approximates atrue solution. This approximation of the solution can serve as astarting point for the numerical optimization and lead to a correctsolution (e.g., the final control points 870) in less time than startingfrom a point with less information. Another benefit from use of aprobabilistic model involves the use of approximations to reliablyachieve higher quality results than would be obtained by starting withless information. For instance, in some settings, the inferred fluencemaps can serve as a lower bound on the expected quality of optimizationof a fluence map.

FMO is a high-dimensional optimization that conventionally uses defaultinitial values of the problem parameters, without regard to the specificpatient's anatomy. FMO is also performed to produce pareto-optimalresults, meaning that the resulting dose distribution satisfies thetarget and OAR constraints such that no constraint can be improvedwithout degrading another. Achieving this high level of accuracyrequires exhaustive computations adjusting beamlet weights in thebeam-normal fluence maps subject to a ranked list of constraints.Existing forms of FMO calculations are therefore time-consuming, oftentaking 10 to 20 minutes for typical prostate cases and much longer formore complex head/neck treatments. Thus, compute times often scale withthe complexity of the plan and the numbers of the constraints. And thiscompletes only the first step of a three-step process.

In various examples, generative machine learning models are adapted toperform FMO to produce fluence map values from input imaging data. Thefollowing examples thus identify ways in which an anatomy-dependentmodel of FMO parameters can be created and trained from a machinelearning model. With use of a trained machine learning model, the FMOparameter calculation can be initialized closer to the end values forthe parameters, reducing the time needed to compute a set of optimalfluence maps.

Additionally, in various examples, the following techniques perform FMOusing machine learning models as part of a measurement, verification, orvalidation process or procedure. For instance, the results from amachine learning model can be used to provide an independent measure ofan FMO program's performance outside of the specific physics andclinical information involved with a patient. Thus, machine learningmodels may provide or evaluate FMO data to serve as a benchmark forfuture radiotherapy plan development.

As a more detailed overview, the following outlines an FMO process,implemented with probabilistic machine learning models, that isperformed in relation to VMAT radiotherapy planning. As discussed above,in VMAT treatments, multiple beams are directed toward the target, andeach beam's cross-sectional shape conforms to the view of the targetfrom that direction, or to a set of segments that all together provide avariable or modulated intensity pattern. Each beam is discretized intobeamlets occupying the elements of a virtual rectangular grid in a planenormal to the beam. The dose is a linear function of beamlet intensitiesor fluence, as expressed with the following equation:

$\begin{matrix}{{d_{i}(b)} = {\sum\limits_{j = 1}^{n}{D_{ij}b_{j}}}} & \left( {{Equation}\mspace{20mu} 1} \right)\end{matrix}$

Where d_(i)(b) is the dose deposited in voxel i from beamlet j withintensity b_(j), and the vector of n beamlet weights is b=(b₁, . . . ,b_(n))^(T). D_(ij) is the dose deposition matrix.

The FMO problem has been solved by multicriteria optimization. Forinstance, Romeijn et al. (“A unifying framework for multi-criteriafluence map optimization models,” Phys Med Biol (49), 1991-2013, 2004)provided the following FMO model formulation:

$\begin{matrix}{{(P):{\min\limits_{x \geq 0}\;{F\left( {d(b)} \right)}}}{{s.t.\mspace{14mu}{\min\limits_{x \geq 0}\;{G_{1}(b)}}} \leq {C_{1}(b)}}\ldots{{{s.t.\underset{x \geq 0}{\mspace{14mu}\min}}\mspace{11mu}{G_{L}(b)}} \leq {C_{L}(b)}}} & \left( {{Equation}\mspace{20mu} 2} \right)\end{matrix}$

where F(d(b)) is a dose objective function, whose minimization issubject to the listed constraints and where the specialized objectives G(b) are subject to dose constraints C(b), and L is the number ofconstraints. The objective F(d(b)) minimizes the difference of the dosebeing calculated d(b) with the prescribed dose P(b):

F(d(b))=Σ_(i) ∥d _(i)(b)−P _(i)(b)∥₂ ²   (Equation 3)

where the sum is over all voxels. In such settings, solutions ofconstrained objectives may be achieved several ways. Pareto optimalsolutions of multicriteria problems can be generated that have theproperty that improving any criterion value is only possible if the atleast one other criterion value deteriorates, and that all the membersof a Pareto optimal family of solutions lie on a Pareto boundary insolution space. By varying the relative weights of the constraints, theplanner can move along through the optimal plans to explore the effectsof the target dose versus organ sparing trade-offs.

An example of FMO operations is evidenced by iCycle or mCycleimplementations; iCycle was originally implemented by researchers atErasmus University Medical Center (Rotterdam, The Netherlands) andre-implemented as mCycle by Elekta Inc. (Stockholm, Sweden; and St.Charles, Mo. USA). iCycle and mCycle FMO performs beam angle and beamprofile (fluence map) optimization based on a wish-list of prioritizedobjectives and constraints. Starting with an empty plan (no beamsselected), optimal beam orientations are selected from predefined set ofinput directions. Iteration i starts with the selection of a candidatebeam for the i-th orientation beam to be added to the plan. Allorientations not yet selected are evaluated one-by-one solving for eachof them a beam profile optimization for the trial i-th beam and all thepreviously-selected beams. Each iterative optimization satisfies all ofthe wishlist objectives and their constraints.

FIG. 9 illustrates examples of FMO operations, providing a comparison ofa conventional FMO process (such as performed by iCycle or mCycle) to amachine-learning-modeled FMO optimization performed with the variousexamples discussed herein. As shown, conventional FMO iterates throughbeam angle selection and profile optimization. Each iteration beginswith the selection of a candidate beam orientation 910 added to the planfollowed by multi-criterial optimization 920 of the beam profiles forthe new beam and all previously-selected beams. The candidate beam withthe best score, representing a best direction 940, is added to the plan.The first optimization stage is complete when additional beams,identified in a search for a new direction 930, fail to improve theoptimization score sufficiently. A second optimization stage 960 isperformed to improve the objectives further if possible. The result ofthis iterative build-up of beams and profiles is a plan 980 that isPareto-optimal with respect to the wishlist objectives and constraints.

In contrast, the machine learning-modeled FMO techniques discussed belowbegins with an estimate of the beam directions and fluence profiles 950learned from a population of clinical plans. This “plan estimate”directly goes to the second optimization stage 960 for refinement withrespect to the wishlist objectives. This avoids the time-consumingbuildup of searching performed by the first optimization stage andachieves shorter times to plan creation, because the machine learningestimate starts closer to the pareto optimum in parameter space than theconventional FMO initialization parameters.

For each iterative beam addition, the target and OAR objectives goalsare guided by a wishlist, which provides a prioritized schedule of hardconstraints and optimization objectives similar to those in Equation(2), above. For example, a wishlist for prostate radiotherapy mayinclude hard constraints (e.g., specifying maximum radiation limits onplanning target volume (PTV) areas, anatomical structures such as therectum, bladder, etc.) and objectives (prioritized goal treatments onPTV areas, anatomical structures such as the rectum, bladder, etc.). Inan example, the wishlist objectives are optimized one at a time in amulticriteria optimization. The highest priority objectives are theprescribed dose to the targets and the dose limits to the mostvulnerable OARs. The lower-priority objectives provide additionalconstraints to achieve the highest therapeutic value possible. At eachstep, the prior optima serve as initial points and prior constraints forsucceeding constraints' optimizations. Further, hard constraints may bedefined to have the highest priority and include hard limits (thatcannot be exceeded) on target areas (PTVs) and the major OARs.Objectives are goals that are attained (if possible) by repeatednumerical optimization of the fluence maps.

In iCycle and mCycle implementations, target dose is optimized byminimizing the logarithmic tumor control probability (LTCP),

$\begin{matrix}{{LTCP} = {\frac{1}{V}{\sum\limits_{i \in V}e^{- {\alpha{({{d_{i}{(b)}} - {P_{i}{(b)}}})}}}}}} & \left( {{Equation}\mspace{20mu} 4} \right)\end{matrix}$

that penalizes underdosage but permits the PTV to be affected by nearbyOARs. In this equation, V is the set of voxels comprising the target PTVand d^(P) is the prescribed dose. a is the cell sensitivityparameter—higher a results in a fewer target voxels with low dose andtherefore a higher fraction of the voxels receiving 95% of theprescribed dose (good PTV coverage). The generalized equivalent uniformdose (gEUD) is a second useful dose function applied to OARs:

$\begin{matrix}{{gEUD} = \left\lbrack {\frac{1}{V}{\sum\limits_{i \in V}\left( {d_{i}(b)} \right)^{a}}} \right\rbrack^{\frac{1}{a}}} & \left( {{Equation}\mspace{20mu} 5} \right)\end{matrix}$

where V here is the set of voxels in the relevant organ and a is aparameter that modulates the dose delivered to that organ.

The wishlist-directed multicriteria optimization actually occurs in twophases. In the first phase, objectives are minimized within theconstraints, proceeding from the first objective to the last objectiveon the wishlist. After each objective minimization, and based on itsresult, the constraint for that objective becomes a new constraint forthe succeeding, lower-priority objectives. Adding the newly-achievedconstraints insures that the lower-priority optimizations do not degradeany higher-priority objectives. Consequently, the lower-priorityobjectives have more constraints than the higher objectives. At the endof the first phase, each objective with a defined goal has eitherattained a value equal to that goal (even if further minimization wouldhave been possible), or attained a value higher than its goal if theminimization constraints prevented the optimization from reaching thatgoal.

In the second phase all objectives are re-minimized to their fullestextent. That means that first phase objectives, apart from the LTCPobjectives, that could have been minimized further, now are minimized tothe greatest extent permitted by relevant constraint set. LTCPobjectives' minimizations are stopped at the defined sufficient value toleave more room for minimization of lower-priority objectives and notneedlessly escalate dose.

The resulting FMO plan (e.g., the pareto-optimal plan 980) is in fact athree-dimensional array of a physical dose in the coordinate frame ofthe target anatomy. Given the anatomy, and an array of fixed beamdirections, the resulting optimal dose distribution is fixed as wellsince the many layers of optimizations are deterministic, at least tothe numerical precision of the computer. Therefore, a set of parametersdefining an optimal 3D dose are the set of optimal beam directions andthe optimal fluence maps, one per beam. Further, if the beams arelimited to a fixed set of angles, the fluence maps alone would definethe 3D dose distribution.

FIG. 10 depicts an example arrangement of the original patient imagery(provided in the 3D CT image set 1001), a projection image 1010 of thetarget anatomy at a 90° gantry angle, and a corresponding fluence mapimage 1020 of planned fluence at that same gantry angle. The projectionimage 1010 of the target anatomy specifically represents a targetplanning target volume (PTV) for treatment and the largest OARs as partof a prostate radiotherapy plan. The fluence map image 1020 specificallyrepresents a 2D array of beamlet weights corresponding to rays throughthe fluence map pixels aimed at the target. The target anatomyprojection image 1010 and the fluence map image 1020 are shown for onehypothetical beam at the gantry angle of 90°, but it will be understoodthat a VMAT treatment plan may have 100-150 beams at different angles,each with its own view of the target anatomy and fluence map.

In an example, various data formatting techniques are applied totraining and inferencing fluence data within a machine learning model,to analyze the projection and fluence map image representations providedin FIG. 10. In practice, the anatomy and fluence or dose data exists in3D rectilinear arrays. The FMO result, however, is the idealized 3D dosedistribution corresponding to the optimal fluence maps. With thetechniques discussed below, 2D fluence map representations can beproduced at geometric planes normal to the beam (in coplanartreatments), in relationship to the linac and patient coordinate systemsanalogous to a beam's-eye-view projections of patient anatomy. Thus,anatomy and fluence maps may be represented as planar projections in acylindrical coordinate system and used to train and draw inferences fromthe machine learning model for FMO.

FIG. 11 first depicts the creation of multiple anatomy projections 1110,1120, 1130 from a 3D volume of CT image data. An equivalent techniquecan be used to produce projections for MR images, and thus it will beunderstood that the following references to CT image data is providedfor purposes of illustration and not limitation. As depicted in FIG. 11,multiple projections of the male pelvic organs are depicted relative toa 3D CT image 1101 of that anatomy, provided with views 1110, 1120, 1130at 0, 45, and 90 degrees respectively (introduced earlier with respectto FIG. 2A). The patient orientation is head-first supine with the headof the patient beyond the top of the projections. The organs at risk(bladder, rectum), the target organs (prostate, seminal vesicles), andtheir encapsulating target volumes (Target1, Target2) are delineated(contoured) and each organ voxel was assigned a constant density value,and densities were summed for voxels in two or more structures.

Projection images through this anatomy about the central axis of the 3DCT volume 1100 and at the assigned densities may be obtained, forexample, using a forward projection capability of the RTK cone beam CTreconstruction toolkit, an open-source cone-beam CT reconstructiontoolkit based on the Insight Toolkit (ITK). In these views, the bladderat 0° is in front of the seminal vesicles (bladder is closest to theviewer) and rotates to the left in the next two views.

FIG. 12 next depicts a set of equivalent-geometry fluence maps,corresponding to the views depicted in FIG. 11. As depicted, 2D fluencemaps are produced from converting an FMO dose distribution volume 1200into a thresholded volume 1202, as a dose distribution from radiotherapybeams (e.g., as depicted in a 2D dose distribution projection 1201),then the thresholded dose intensity (as shown in 1205) is converted intoprojection map-images (e.g., in 2D fluence map projections 1210, 1220,and 1230). For instance, projections from a 3D ideal fluencedistribution may be established using the forward projection capabilityof the RTK cone beam CT reconstruction toolkit and the ITK toolkit. Theproduced projection views correspond to those views shown in FIG. 11:fluence map 1210 corresponding to 0-degree projection view 1110; fluencemap 1220 corresponding to 45 degree projection view 1120; fluence map1230 corresponding to 90 degree projection view 1130.

FIG. 13 further depicts, in each respective row, a set of 2D anatomyprojections 1310, corresponding 2D fluence maps 1320, and superimposed2D fluence maps 1330 that show fluence maps overlaid on anatomyprojections. FIG. 13 further depicts, in each column, these projectionsand maps at linac gantry angles 90° (arrangement 1340), 120°(arrangement 1350), 150° (arrangement 1360), and 180° (arrangement1370), respectively. Through the use of projection transformations, the3D voxel data can be accurately be represented in a format compatiblewith the geometry of the treatment.

In various examples, these projections may be used for training machinelearning models to produce a prediction of fluence map or planparameters. Specifically, the following approaches discuss a trainedmachine learning model that predicts fluence map or plan parametersgiven only a new patient's images and relevant radiotherapy anatomystructures, such as OARs and treatment targets.

In an example, the prediction is made using probabilistic models ofplans learned from populations of existing optimal fluence plans. Thenew patient data combined with the model enables a prediction of afluence plan that serves as the starting point for direct apertureoptimization. Among other benefits, this enables the time to refine theplan to clinical quality to be reduced.

The probabilistic models may be constructed as follows. Representing theanatomy data as a kind of random variable X, and the fluence planinformation as random variable Y. Bayes' Rule states that theprobability of predicting a plan Y given a patient X, p(Y|X), isproportional to the conditional probability of observing patient X giventhe training plans, Y, p(X|Y) and the prior probability of the trainingplans p(Y), or:

p(Y|X)∝p(X|Y)p(Y)   (Equation 6)

Bayesian inference predicts a plan Y* for a novel patient X* where theconditional probability p(Y*|X*) is drawn from the training posteriordistribution p(Y|X). In practice, the novel anatomy X* is input to thetrained network that then generates an estimate of the predicted plan Y*from the stored model p(Y|X).

FIG. 14 depicts a schematic of the deep learning procedure to train amodel to predict fluence maps. In an example, the training data includespairs of 3D imaging projections 1410 and 3D stacks of fluence maps 1420from the same data source (e.g., a same patient). Training produces themodel

1430 from which an estimate

can be inferred. The estimate is itself a 3D data volume 1440 with thesame size and shape as the input anatomy and fluence data volumes. Thatestimate can be translated into a functional set of fluences and used asa warm start to accelerate FMO.

The plan posterior models p(Y|X) are built by training convolutionalneural networks with pairs of known data (anatomy, plan; X, Y) in anoptimization that minimizes network loss functions and simultaneouslydetermines the values of the network layer parameters Θ. These neuralnetwork parameter values embed the posterior model p(Y|X) as p(Y|X; Θ).Once trained, the network can infer a plan for a new anatomy by theBayes analysis described above. Neural network performance isestablished by comparing the inferred plans for test patients not usedfor training with those same test patients' original clinical plans—thebetter the neural network, the smaller the differences between the setsof plans.

In various examples, various forms of machine learning models may beimplemented by artificial neural networks (NNs). At its simplestimplementation, a NN consists of an input layer, a middle or hiddenlayer, and an output layer. Each layer consists of nodes that connect tomore than one input node and connect to one or more output nodes. Eachnode outputs a function of the sum of its inputs x=(x₁, . . . , x_(n)),y˜σ(w^(T)x+β), where w is the vector of input node weights and β is thelayer bias and the nonlinear function σ is typically a sigmoidalfunction. The parameters Θ=(w, β) are the realization of the modellearned to represent the relationship Y=ƒ(X; Θ). The number of inputlayer nodes typically equals the number of features for each of a set ofobjects being sorted into classes, and the number of output layer nodesis equal to the number of classes. For regression, the output layertypically has a single node that communicates the estimated or probablevalue of the parameter.

A network is trained by presenting it with object features where theobject's class or parameter value is known and adjusting the nodeweights w and biases β to reduce the training error by working backwardfrom the output layer to the input layer—an algorithm calledbackpropagation. The training error is a normed difference ∥y−ƒ(x)∥between the true answer y and the inference estimate ƒ(x) at any stageof training. The trained network then performs inference (eitherclassification or regression) by passing data forward from input tooutput layer, computing the nodal outputs σ(w^(T)x+β) at each layer.

Neural networks have the capacity to discover general relationshipsbetween the data and classes or regression values, including non-linearfunctions with arbitrary complexity. This is relevant to the problem ofradiotherapy dose prediction, or treatment machine parameter prediction,or plan modelling, since the shape or volume overlap relationships oftargets and organs as captured in the dose-volume histogram and theoverlap-volume histogram are highly non-linear and have been shown to beassociated with dose distribution shape and plan quality.

Modern deep convolutional neural networks (CNNs) have many more layers(are much deeper) than early NNs—and may include dozens or hundreds oflayers, each layer composed of thousands to hundreds of thousands ofnodes, with the layers arranged in complex geometries. In addition, theconvolution layers map isomorphically to images or any other data thatcan be represented as multi-dimensional arrays and can learn featuresembedded in the data without any prior specification or feature design.For example, convolution layers can locate edges in pictures, ortemporal/pitch features in sound streams, and succeeding layers findlarger structures composed of these primitives. In the past half-dozenyears, some CNNs have approached human performance levels on canonicalimage classification tests—correctly classifying pictures into thousandsof classes from a database of millions of images.

CNNs are trained to learn general mappings ƒ: X→Y between data in sourceand target domains X, Y, respectively. Examples of X include images ofpatient anatomy or functions of anatomy conveying structuralinformation. Examples of Y could include maps of radiation fluence ordelivered dose, or maps of machine parameters superposed onto the targetanatomy X. As indicated in FIG. 14, pairs of matched, known X, Y datamay be used to train a CNN. The CNN learns a mapping or function ƒ(X; Θ)of both anatomy and network parameters Θ=(θ₁, . . . , θ_(n))^(T) whereθ_(i)={w_(i), β_(i)} are the parameters for the i-th layer. Trainingminimizes a loss function

(Θ) over the mapping ƒ and a ground truth or reference plan parameter Ŷ

$\begin{matrix}{{{\mathcal{L}\left( \theta^{*} \right)} = {\underset{\Theta}{\arg\;\min}\left\lbrack {{{{f\left( {X;\Theta} \right)} - \hat{Y}}}_{K} + {\lambda{\Theta }_{L}}} \right\rbrack}},K,{L \in \left\{ {1,2} \right\}}} & \left( {{Equation}\mspace{20mu} 7} \right)\end{matrix}$

where the first term minimizes the difference between the networkestimated target ƒ(X; Θ) and the reference property Ŷ and the secondterm minimizes the variation of the values of the Θ. Subscripts K, Lspecify the norm. The L2 norm (K, L=2) is globally convex but producesblurred estimates of Y while the L1 norm (K, L=1) encourage sharperestimates. Network performance typically dictates what combination ofnorms are useful.

FIG. 15A depicts a schematic of a U-Net deep convolutional neuralnetwork (CNN). Specifically, this schematic depicts the U-Net deep CNNmodel adapted for generating a fluence map representation in agenerative arrangement, such as to provide a generative model adaptedfor the techniques discussed herein. Shown are a pair of input imagesrepresenting target anatomy constraints (top image) and a radiotherapytreatment X-ray fluence representation corresponding to that targetanatomy (bottom image), provided in an input training set 1510 to trainthe network. The output is a predicted fluence map representation 1540,inferred for a target image. The input training set 1510 may includeindividual pairs of input images that are projected from a 3D anatomyimaging volume and 3D fluence volume; these individual pairs of inputimages may comprise individual images that are projected at relevantbeam angle used for treatment with a radiotherapy machine. The outputdata set, provided in the fluence map representation 1540, is arepresentation that may comprise individual output images or a 3Dfluence volume.

A U-Net CNN creates scaled versions of the input data arrays on theencoding side by max pooling and re-combines the scaled data withlearned features at increasing scales by transposed convolution on theencoding side to achieve high performance inference. The blackrectangular blocks represent combinations of convolution/batchnormalization/rectified linear unit (ReLU) layers; two or more are usedat each scale level. The blocks' vertical dimension corresponds to theimage scale (S) and the horizontal dimension is proportional to thenumber of convolution filters (F) at that scale. Equation 7 above is atypical U-Net loss function.

The model shown in FIG. 15A depicts an arrangement adapted forgenerating an output data set (output fluence map representation images1540) based on an input training set 1510 (e.g., paired anatomy imagesand fluence map representation images). The name derives from the “U”configuration, and as is well understood, this form of CNN model canproduce pixel-wise classification or regression results. In some cases,a first path leading to the CNN model includes one or more deformableoffset layers and one or more convolution layers including convolution,batch normalization, and an activation such as the rectified linear unit(ReLU) or one of its variants.

The left side of the model operations (the “encoding” operations 1520)learns a set of features that the right side (the “decoding” operations1530) uses to reconstruct an output result. The U-Net has n levelsconsisting of conv/BN/ReLU (convolution/batch normalization/rectifiedlinear units) blocks 1550, and each block has a skip connection toimplement residual learning. The block sizes are denoted in FIG. 15A by“S” and “F” numbers; input images are S×S in size, and the number offeature layers is equal to F. The output of each block is a pattern offeature responses in arrays the same size as the images.

Proceeding down the encoding path, the size of the blocks decreases by ½or 2⁻¹ at each level while the size of the features by conventionincreases by a factor of 2. The decoding side of the network goes backup in scale from S/2″ while adding in feature content from the left sideat the same level; this is the copy/concatenate data communication. Thedifferences between the output image and the training version of thatimage drives the generator network weight adjustments bybackpropagation. For inference, or testing, with use of the model, theinput would be a single projection image or collection of multipleprojection images of radiotherapy treatment constraints (e.g., atdifferent beam or gantry angles) and the output would be graphicalfluence map representation images 1540 (e.g., one or multiple graphicalimages corresponding to the different beam or gantry angles).

The representation of the model of FIG. 15A specifically illustrates thetraining and prediction of a generative model, which is adapted toperform regression rather than classification. FIG. 15B illustrates anexemplary CNN model adapted for discriminating a synthetic fluence maprepresentation(s) according to the present disclosure. As used herein, a“synthetic” image refers to a model-generated image, and thus“synthetic” is used interchangeably herein with the terms “estimated”,“computer-simulated”, or “computer-generated”. The discriminator networkshown in FIG. 15B may include several levels of blocks configured withstride-2 convolutional layers, batch normalization layers and ReLUlayers, and separated pooling layers. At the end of the network, therewill be one or a few fully connection layers to form a 2D patch fordiscrimination purposes. The discriminator shown in FIG. 15B may be apatch-based discriminator configured to receive input synthetic fluencemap representation images (e.g., generated from the generator shown inFIG. 15A), classify the image as real or fake, and provide theclassification as output detection results 1570.

In an example, the present FMO modeling techniques may be generatedusing a specific a type of CNN, generative adversarial networks (GANs),to predict fluence plan parameters (a fluence map) from radiotherapytreatment constraints of new patient anatomy. In a further example, acycle-consistent GAN may be used to predict fluence plan parameters fromnew patient anatomy. The following provides an overview of relevant GANtechnologies.

Generative adversarial networks are generative models (generateprobability distributions) that learn a mapping from random noise vectorz to output image y as G: z→y. Conditional adversarial networks learn amapping from observed image x and random noise z as G: {x, z}→y. Bothadversarial networks consist of two networks: a discriminator (D) and agenerator (G). The generator G is trained to produce outputs that cannotbe distinguished from “real” or actual training images by an adversarialtrained discriminator D that is trained to be maximally accurate atdetecting “fakes” or outputs of G.

The conditional GAN differs from the unconditional GAN in that bothdiscriminator and generator inferences are conditioned on an exampleimage of the type X in the discussion above. The conditional GAN lossfunction is expressed as:

_(cGAN)(G, D)=

_(x,y˜p) _(data) _((x,y))[log(D(x,y))]

E _(x˜p) _(data) _((x), z˜p) _(z) _((z))[log(1−D(x, G(x,z)))]  (Equation 8)

where G tries to minimize this loss against an adversarial D that triesto maximize it, or,

$\begin{matrix}{G^{*} = {\underset{G}{\arg\;\min}\;{\underset{D}{\;\max}\;{\mathcal{L}_{cGAN}\left( {G,D} \right)}}}} & \left( {{Equation}\mspace{20mu} 9} \right)\end{matrix}$

In addition, one wants the generator G to minimize the differencebetween the training estimates and the actual training ground truthimages,

_(L1)(G)=

_(x,y˜p) _(data) _((x,y), z˜p) _(z) _((z)) [∥−G(x,z)∥₁]  (Equation 10)

so, the complete loss is the λ-weighted sum of two losses:

$\begin{matrix}{G^{*} = {{\underset{G}{\arg\;\min}\;\underset{D}{\;\max}\;{\mathcal{L}_{cGAN}\left( {G,D} \right)}} + {\lambda\;{\mathcal{L}_{L\; 1}(G)}}}} & \left( {{Equation}\mspace{20mu} 11} \right)\end{matrix}$

In an example, the generator in the conditional GAN may be a U-Net.

Consistent with examples of the present disclosure, the treatmentmodeling methods, systems, devices, and/or processes based on suchmodels include two stages: training of the generative model, with use ofa discriminator/generator pair in a GAN; and prediction with thegenerative model, with use of a GAN-trained generator. Various examplesinvolving a GAN and a CycleGAN for generating fluence map representationimages are discussed in detail in the following examples. It will beunderstood that other variations and combinations of the type of deeplearning model and other neural-network processing approaches may alsobe implemented with the present techniques. Further, although thepresent examples are discussed with reference to images and image data,it will be understood that the following networks and GAN may operatewith use of other non-image data representations and formats.

FIG. 16A illustrates a data flow for training and use of a GAN adaptedfor generating a fluence plan parameters (a fluence map representation)from a received set of projection images that represents a view of ananatomy of a subject image. For instance, the generator model 1632 ofFIG. 16A, which is trained to produce a trained generator model 1660,may be trained to implement the processing functionality provided aspart of the image processor 114 in the radiotherapy system 100 of FIG.1.

Accordingly, a data flow of the GAN model usage 1650 (prediction orinference) is depicted in FIG. 16A as the provision of new patient data1670 (e.g., a projection image that represents radiotherapy treatmentconstraints in a view of an anatomy of a subject input images from anovel patient) to a trained generator model 1660, and the use of thetrained generator model 1660 to produce a prediction or estimate of agenerator output (images) 1634 (e.g., synthetic graphical fluence maprepresentation images corresponding to the input projection image thatrepresents a view of an anatomy of a subject image). A projection imagecan be generated from one or more CT or MR images of a patient anatomyrepresenting a view of the anatomy from a given beam position (e.g., atan angle of the gantry) or other defined positions.

GANs comprise two networks: a generative network (e.g., generator model1632) that is trained to perform classification or regression, and adiscriminative network (e.g., discriminator model 1640) that samples thegenerative network's output distribution (e.g., generator output(images) 1634) or a training fluence map representation image from thetraining images 1623 and decides whether that sample is the same ordifferent from the true test distribution. The goal for this system ofnetworks is to drive the generator network to learn the ground truthmodel as accurately as possible such that the discriminator net can onlydetermine the correct origin for generator samples with 50% chance,which reaches an equilibrium with the generator network. Thediscriminator can access the ground truth but the generator onlyaccesses the training data through the response of the detector to thegenerator's output.

The data flow of FIG. 16A also illustrates the receipt of training input1610, including various values of model parameters 1612 and trainingdata 1620 (with such training images 1623 including a set of projectionimages that represent different views of an anatomy of subject patientimaging data paired with real graphical fluence map representationscorresponding to the patient imaging data at the different views, andconditions or constraints 1626. These conditions or constraints 1626(e.g., one or more radiotherapy treatment target areas, one or moreorgans at risk areas, etc.) may be indicated directly in the anatomyimages themselves (e.g., as shown with projection image 1010), orprovided or extracted as a separate data set. The training input 1610 isprovided to the GAN model training 1630 to produce a trained generatormodel 1660 used in the GAN model usage 1650.

As part of the GAN model training 1630, the generator model 1632 istrained on real training fluence map representation images andcorresponding training projection images that represent views of ananatomy of a subject image pairs 1622 (also depicted in FIG. 16A as1623), to produce and map segment pairs in the CNN. In this fashion, thegenerator model 1632 is trained to produce, as generator output (images)1634, computer-simulated (estimated or synthetic) images of fluence maprepresentations and fluence values based on an input map. Thediscriminator model 1640 decides whether a simulated fluence maprepresentation image or images is from the training data (e.g., thetraining or true fluence map representation images) or from thegenerator (e.g., the estimated or synthetic fluence map representationimages), as communicated between the generator model 1632 and thediscriminator model 1640. The discriminator output 1636 is a decision ofthe discriminator model 1640 indicating whether the received image is asimulated image or a true image and is used to train the generator model1632. In some cases, the generator model 1632 is trained utilizing thediscriminator on the generated images and is further trained based oncycle-consistency loss information. This training process results inback-propagation of weight adjustments 1638, 1642 to improve thegenerator model 1632 and the discriminator model 1640.

During training of generator model 1632, a batch of training data can beselected from the patient images (indicating radiotherapy treatmentconstraints) and expected results (fluence map representations). Theselected training data can include at least one projection image ofpatient anatomy representing a view of the patient anatomy from a givenbeam/gantry angle and the corresponding training or real fluence maprepresentations image at that given beam/gantry angle. The selectedtraining data can include multiple projection images of patient anatomyrepresenting views of the same patient anatomy from multiple equallyspaced or non-equally spaced angles (e.g., at gantry angles, such asfrom 0 degrees, from 15 degrees, from 45 degrees, from 60 degrees, from75 degrees, from 90 degrees, from 105 degrees, from 120 degrees, from135 degrees, from 150 degrees, from 165 degrees, from 180 degrees, from195 degrees, from 210 degrees, from 225 degrees, from 240 degrees, from255 degrees, from 270 degrees, from 285 degrees, from 300 degrees, from315 degrees, from 330 degrees, from 345 degrees, and/or from 360degrees) and the corresponding training fluence map representation imageand/or machine parameter data at those different equally-spaced ornon-equally spaced gantry angles.

Thus, in this example, data preparation for the GAN model training 1630requires fluence map representation images that are paired withprojection images that represent views of an anatomy of subject images(these may be referred to as training projection images that represent aview of an anatomy of a subject image at various beam/gantry angles).Namely, the training data includes paired sets of fluence maprepresentation images at the same gantry angles as the correspondingprojection images. In an example, the original data includes pairs ofprojection images that represents a view of an anatomy of a subject atvarious beam/gantry angles and corresponding fluence map representationsof fluence at the corresponding beam/gantry angles that may beregistered and resampled to a common coordinate frame to produce pairsof anatomy-derived images. The training data can include multiple ofthese paired images for multiple patients at any number of differentbeam/gantry angles. In some cases, the training data can include 360pairs of projection images and fluence map representation images, onefor each angle of the gantry for each training patient.

The expected results can include estimated or synthetic graphicalfluence map representations of fluence outcomes, that can be furtheroptimized and converted into control points. Such control points may beconverted and optimized into control points for generating a beam shapeat the corresponding beam/gantry angle to define the delivery ofradiation treatment to a patient. The control points or machineparameters can include at least one beam/gantry angle, at least onemulti-leaf collimator leaf position, and at least one aperture weight orintensity, based on the specifications of the fluence map.

In detail, in a GAN model, the generator (e.g., generator model 1632)learns a distribution over the data x, p_(G)(x), starting with noiseinput with distribution p_(Z)(z) as the generator learns a mapping G (z;θ_(G)): p_(Z)(z)→p_(G)(x) where G is a differentiable functionrepresenting a neural network with layer weight and bias parametersθ_(G). The discriminator, D(x; θ_(D)) (e.g., discriminator model 1640),maps the generator output to a binary scalar {true, false}, decidingtrue if the generator output is from actual data distributionp_(data)(x) and false if from the generator distribution p_(G)(x). Thatis, D(x) is the probability that x came from p_(data)(x) rather thanfrom p_(G)(x). In another example, paired training data may be utilizedin which, for instance, Y is conditioned (dependent) on X. In suchcases, the GAN generator mapping is represented by G(y|x; θ_(G)): X→Yfrom data domain X where data x∈X represents the anatomy projectionimages and domain Y where data y∈Y represents the fluence maprepresentation values corresponding to x. Here an estimate for anfluence map representation value is conditioned on its projection.Another difference from the straight GAN is that instead of a randomnoise z input, the projection image x is the generator input. For thisexample, the setup of the discriminator is the same as above. Ingeneral, the generator model 1632 and the discriminator model 1640 arein a circular data flow, where the results of one feed into the other.The discriminator takes either training or generated images and itsoutput is used to both adjust the discriminator weights and to guide thetraining of the generator network.

Another useful extension of a GAN is the CycleGAN, which is describedbelow in connection with FIG. 16B. FIG. 16B illustrates training and useof CycleGAN 1631 for generating a collection of fluence maprepresentation images (e.g., a collection of synthetic or estimatedfluence map representation images, projected at radiotherapy beamangles) from a received collection of projection images (e.g., acollection of projection images of anatomy indicating radiotherapytreatment constraints, projected at the radiotherapy beam angles)according to some examples of the disclosure. CycleGAN 1631 includes afirst generator model 1635, second generator model 1637, a firstdiscriminator model 1639A, and a second discriminator model 1639B. Thefirst generator model 1635 includes deformable offset layers andconvolution blocks and the second generator model 1637 includesdeformable offset layers and convolution blocks. These two models 1635and 1637 may each be an implementation of generator model 1632 (e.g., inFIG. 16A, as regression-type DCNN models), and first discriminator model1639A and second discriminator model 1639B may each be an implementationof discriminator model 1640 (e.g., as classification-type DCNN models).CycleGAN 1631 may be divided into two portions, first portion 1633A andsecond portion 1633B.

The convolution blocks of each generator model 1635 and 1637 may betrained together or separate from training of the other generator anddiscriminator models. Specifically, the convolution blocks of thegenerator models 1635 and 1637 are trained to obtain the correct weightsto perform their function. The deformable offset layers may each betrained to coordinate offsets, resample, and perform interpolation. Thedeformable offset layers may be trained together or separate fromtraining of the generator and discriminator models. The effect of theseoffset layers changes the original regular sampling grids from upperconvolutional blocks, introduces coordinate offsets, and resamples theimages using interpolation. The deformable offset layers mayalternatively or in addition be implemented using a spatial transformer,other types of convolutional layers, and/or any other module that canstore deformed structure information for an image. The number of offsetlayers in the deformable offset layers may vary based on image size, thenumber of down-sampling convolutional layers, and other factors.

In an example, in first portion 1633A, the first generator model 1635may be trained to receive a training collection of projection images1623A (which may include anatomical projection images, one of imagepairs 1622) and generate a respective synthetic first collection offluence map representation images as first generation results 1636A. Thefirst generator model 1635 is referred to as G^(proj2fluence).

First generation results 1636A may be provided to the firstdiscriminator model 1639A. The first discriminator model 1639A mayclassify the synthetic collection of fluence map representation imagesas a real collection of fluence map representation training images or asimulated collection of fluence map representation training images andprovide the classification as detection results 1644A. The firstgeneration results 1636A and detection results 1644A may be fed back tothe first generator model 1635 and first discriminator model 1639A toadjust weights implemented by the first generator model 1635 and firstdiscriminator model 1639A. For example, first generation result 1636A(e.g., a collection of fluence map representation images generated byfirst generator model 1635) and detection results 1644A may be used tocalculate adversarial losses.

The first generation results 1636A (e.g., the synthetic collection offluence map representation images) may also be concurrently provided tothe second generator model 1637. The second generator model 1637 mayreceive first generation results 1636A and generate a respectivesimulated collection of anatomical projection images as outputs. Thesimulated collection of anatomical projection images may be referred toas a cycle collection of anatomical projection images 1641 and may beused to compute cycle losses to adjust weights of first/second generatormodel 1635/1637. The second generator model 16that generates the firstcycle collection of anatomical projection images 1641 is referred to asG^(fluence2proj).

In an example, in the second portion 1633B, the second generator model1637 may be trained to receive a real collection of training fluence maprepresentation images 1623B (which may include one of image pairs 1622)and to generate a respective synthetic collection of anatomicalprojection images (a synthetic or simulated collection of anatomicalprojection images) as first generation results 1636B. The secondgenerator model 1637 that generates the first generation results 1636Bis the same generator as that used in the first portion 1633A.

First generation results 1636B may be provided to the seconddiscriminator model 1639B. The second discriminator model 1639B mayclassify the synthetic collection of anatomical projection images as areal collection of anatomical projection training images or a simulatedcollection of anatomical projection training images and provide theclassification as detection results 1644B. The first generation results1636B and detection results 1644B may be fed back to the secondgenerator model 1637 and the second discriminator model 1639B to adjustweights implemented by second generator model 1637 and seconddiscriminator model 1639B. For example, first generation result 1636B(e.g., a synthetic collection of anatomical projection images generatedby second generator model 1637) and the detection results 1644B may beused to calculate adversarial losses.

First generation results 1636B (e.g., synthetic collection of anatomicalprojection images) may also be concurrently provided to the firstgenerator model 1635. The first generator model 1635 may receive firstgeneration results 1636B and generate respective cycle-fluence maprepresentation images 1643 as outputs. The cycle-fluence maprepresentation images 1643 may be used to compute cycle losses to adjustweights of first/second generator model 1635/1637. The first generatormodel 1635 that generates the cycle-fluence map representation images1643 is the same generator as that used in the first portion 1633A, andthe second generator model 1637 that generates the cycle-fluence maprepresentation images 1643 is the same generator as that used in thefirst portion 1633A.

In some examples, “adversarial losses” may account for theclassification losses for the first and second discriminator models1639A and 1639B. First and second discriminator models 1639A and 1639Bmay classify whether the synthetic images have similar distribution astrue images or not. For cycle-consistency losses, the losses arecalculated between each pair of true collection of projection images andcycle-collection of projection images, and each pair of true fluence maprepresentation images and cycle-fluence map representation images,respectively. For example, a first loss may be calculated between acollection of projection training images 1623A and cycle-collection ofprojection images 1641 and a second loss may be calculated between realtraining collection of fluence map representation images 1623B andcollection of cycle-fluence map representation images 1643. Thecycle-collection of projection images 1641 and cycle-fluence maprepresentation images 1643 may both be obtained by doing forward andbackward cycles. Each pair of true collection of projection images 1623Aand cycle-collection of projection images 1641 may be in the samecollection of projection images domain, and each pair of real trainingfluence map representation images 1623B and cycle-fluence maprepresentation images 1643 may be in the same graphical fluence maprepresentation image domain. The CycleGAN 1631 may accordingly rely on awhole pool (or a plurality) of true or real projection training images1623A and a whole pool (or a plurality) of real training fluence maprepresentation images 1623B to produce synthetic fluence maprepresentation images (collection of fluence map representation images),synthetic collection of projection images, cycle-collection ofprojection images 1641, and cycle-fluence map representation images1643. Based on “adversarial losses” and “cycle-consistency losses,”CycleGAN 1631 may produce sharp synthetic fluence map representationimages, which have similar image resolution as real fluence maprepresentation images.

In some examples, a processor (e.g., of radiotherapy system 100) mayapply image registration to register real fluence map representationtraining images to a training collection of projection images. This maycreate a one-to-one corresponding relationship between projection imagesat different angles (e.g., beam angles, gantry angles, etc.) and fluencemap representation images at each of the different angles in thetraining data. This relationship may be referred to as paired or a pairof projection images and fluence map representation images.

In some implementations, CycleGAN 1631 may be implemented to generate acollection of fluence map representation images in accordance with anobjective function that includes an adversarial loss term and a cycleconsistency loss term. The CycleGAN network has two separate adversariallosses. Similar to the conditional GAN, the mapping G:X→Y and itsassociated discriminator Dy gives loss represented by:

_(GAN)(G, D _(y))=

_(y˜p) _(data) _((y))[log(D _(y)))]+E _(x˜p) _(data) _((x))[log(1−D_(y)(G(x)))]  (Equation 12)

With the CycleGAN, a network's forward cycle-consistencyx→G(x)→F(G(x))≈x and the network's backward cycle-consistencyy→F(y)→G(F(y))≈y. The adversarial losses in the network (e.g., usingfirst/second generator models 1635/1637 and first/second discriminatormodels 1639A/1639B) are captured in the cycle-consistency losses as theL1 norms,

_(cyc)(G, F)=

_(x˜p) _(data) _((x)) [∥F(G(x))−x∥ ₁ ]+E _(y˜p) _(data) _((y))[∥G(F(y))−y∥ ₁]  (Equation 13)

Additionally, the identity loss regularizes the generator to be near anidentity mapping when real samples of the target domain Y are input:

_(identity)(G, F)=

_(y˜p) _(data) _((y)) [∥G(y)−y∥ ₁]+

_(x˜p) _(data) _((x)) [∥F(x)−x∥ _(1])  (Equation 14)

Thus, the full loss function for the cycle-consistent GAN is

(G, ƒ, D _(X) , D _(Y))=

_(GAN)(G, D _(Y))+

_(GAN)(F, D _(X))+λ_(cyc)

_(cyc)(G, F)+λ_(identity)

_(identity)(G, F)   (Equation 15)

where D_(X) is the first discriminator model which determines whetherone image is a true collection of fluence map representation images or asynthetic collection of fluence map representation images. D_(Y) is thesecond discriminator model which determines whether one image is a truecollection of projection images or a synthetic collection of projectionimages.

The preceding examples provide an example of how a GAN, a conditionalGAN, or CycleGAN may be trained based on a collection of fluence maprepresentation images and collection of projection image pairs,specifically from image data in 2D or 3D image slices in multipleparallel or sequential paths. It will be understood that the GAN,conditional GAN, or CycleGAN may process other forms of image data(e.g., 3D, or other multi-dimensional images) or representations of thisdata including in non-image format. Further, although only grayscale(including black and white) images are depicted by the accompanyingdrawings, it will be understood that other image formats and image datatypes may be generated and/or processed by the GAN.

FIG. 17 illustrates an example flowchart 1700 of a method for training aneural network model, such as a model that will be trained forgenerating a fluence map using the techniques and constraints discussedabove. It will be apparent that additional operations, or a variation inthe sequence of operations, may be implemented within this method.

At operation 1710, operations are performed to obtain training anatomyprojection images, and at operation 1720, operations are performed toobtain training fluence map projection images. In this training scenarioof flowchart 1700, pairs of anatomy projection images and fluence mapsmay be obtained from a plurality of human subjects, such that eachcorresponding pair of projection images and fluence maps is providedfrom a same human subject. Further, the corresponding pairs of theanatomy projection images and the fluence maps used for training theneural network model may be obtained for each beam angle of aradiotherapy treatment machine The operations 1710, 1720 may alsoinvolve other aspects of identifying, extracting, projecting, andmodifying the projection images and the fluence maps, as suggestedabove.

The flowchart 1700 proceeds to operation 1730, to perform and supervisetraining of the model. In various examples, the model is implemented asa neural network, and the neural network model may be a generative modelof a generative adversarial network (GAN). Such a GAN may include atleast one generative model and at least one discriminative model, wherethe at least one generative model and the at least one discriminativemodel correspond to respective generative and discriminativeconvolutional neural networks. In still further examples, the GANcomprises a conditional adversarial network (cGAN) or a cycle-consistentgenerative adversarial network (CycleGAN). In operations 1740-1750discussed below, operations are performed for GAN training; inoperations 1755-1775, other operations are performed for CycleGANtraining.

In an example, operations for GAN training (operations 1740-1750)involve using the GAN to train the generative model using adiscriminative model. For instance, this may involve establishing theneural network parameter values using adversarial training between thediscriminative model and the generative model, for learning the valuesby the generative model and the discriminative model. Such adversarialtraining may involve training the generative model to generate a firstestimated fluence map at a first beam angle from a projection image thatrepresents a view of a training subject anatomy from the first beamangle (operation 1740), training the discriminative model to classifythe first estimated fluence map as an estimated or as a real trainingfluence map projection image (operation 1745), and using an output ofthe generative model for training the discriminative model and an outputof the discriminative model for training the generative model (operation1750).

In an example, operations for CycleGAN training (operations 1755-1775)involve using two sets of models in a GAN arrangement, involving a firstdiscriminative model and a first generative model (trained withoperations 1755, 1760, that correspond to operations 1740, 1745), and asecond generative model and a second discriminative model (trained withoperations 1765, 1770). Specifically, the second generative model istrained for processing, from a given pair of the pairs of anatomyprojection images and fluence maps, a given fluence map at a given beamangle as an input, and generating an estimated anatomy projection imagethat represents a view of a subject anatomy from the given beam angle asan output. The second discriminative model is trained to classify theestimated anatomy projection image as an estimated or as a real anatomyprojection image.

In an example, adversarial training for the first portion of theCycleGAN (the first generative and first discriminative models,corresponding to operations 1755, 1760) involves: obtaining a set oftraining anatomy projection images representing different views of apatient anatomy from prior treatments that are paired with a set oftraining fluence maps corresponding to each of the different views, eachof the training fluence maps being aligned with a respective one of thetraining anatomy projection images; inputting the set of traininganatomy projection images to the first generative model; and outputtinga set of estimated fluence maps from the first generative model;inputting the set of estimated fluence maps to the first discriminativemodel, and classifying the set of estimated fluence maps with the firstdiscriminative model as an estimated or as a real set of fluence maps;and inputting the set of fluence maps to the second generative model,and generating a set of estimated anatomy projection images, tocalculate the cycle-consistency losses. In an example, adversarialtraining for the second portion of the CycleGAN (the second generativeand second discriminative models, corresponding to operations 1765 and1770) involves: inputting the set of training fluence maps correspondingto each of the different views to the second generative model, andoutputting a set of generated anatomy projection images from the secondgenerative model; inputting the set of generated anatomy projectionimages to the second discriminative model, and classifying the set ofgenerated anatomy projection images as an estimated or real set ofanatomy projection images; and inputting the set of anatomy projectionimages to the first generative model to generate a set of estimatedfluence maps to calculate the cycle-consistency losses. From thisadversarial training, cycle-consistency losses may be calculated andconsidered, to improve the training of the first generative model andthe CycleGAN overall (operation 1775).

The flowchart 1700 concludes at operation 1780 with providing trainedgenerative model (from the GAN, CycleGAN, or other training arrangement)for use with patient anatomy projection image(s), and in the radiologytreatment planning processes described herein.

FIG. 18 illustrates an example of a method in a flowchart 1800 for usinga trained neural network model, for determining a fluence map, based onthe techniques discussed above. Additional operations, or a variation inthe sequence of operations, may be implemented within this method,particularly when implemented as part of radiotherapy planning ortreatment operations.

The flowchart 1800 begins with operation 1810, to obtain athree-dimensional set of image data corresponding to a subject ofradiotherapy treatment. For instance, this may be performed by obtainingdata from the use of imaging modalities (e.g., CT, MRI) that image asubject patient. The flowchart 1800 continues with operation 1820, toobtain the radiotherapy treatment constraints for the subject. Forinstance, this may be defined with the definition of target dose areas,organ at risk areas, as part of a radiotherapy treatment planningprocess. The flowchart 1800 continues with operation 1830, to generatethree-dimensional image data indicating radiotherapy treatmentconstraints (e.g., the target dose areas, organ at risk areas). In anexample, the input data provided with a trained model, referenced inoperation 1860, below, is image data that indicates one or more targetdose areas and one or more organs-at-risk areas in the anatomy of thesubject.

The flowchart 1800 continues with operations to perform forwardprojection on three-dimensional image data (at operation 1840) andgenerate a projection image of the subject anatomy for each radiotherapybeam angle (at operation 1850). In an example, each anatomy projectionimage provides a view of the subject from a respective beam angle of theradiotherapy treatment, such as angles that correspond to each gantryangle used by a radiotherapy treatment machine.

The flowchart 1800 continues at operation 1860 with the use of a trainedneural network model (e.g., trained according to the method indicated inflowchart 1700) to generate one or more fluence maps. In an example, theneural network model is trained with corresponding pairs of anatomyprojection images and fluence maps provided from among multiple humansubjects (e.g., as discussed with reference to FIG. 17). The flowchart1800 concludes at operation 1870 with the production of athree-dimensional fluence map representation, such as provided fromgeneration of multiple two-dimensional fluence maps generated by thetrained model. In an example, the generated (estimated) fluence maps andthe training of the model is provided for each radiotherapy beam angleused in radiotherapy treatment.

FIG. 19 provides a flowchart 1900 illustrating overall exampleoperations of a processing system (e.g., the image processing device 112or other computing system) coordinating a radiotherapy treatment andplanning method, according to various examples. As discussed above,additional operations, or a variation in the sequence of operations, maybe implemented within this method, particularly when implemented as partof radiotherapy planning or treatment operations.

At operation 1910, the method begins by obtaining three-dimensionalimage data, including radiotherapy constraints, that correspond to asubject for radiotherapy treatment. As indicated above, such image datamay indicate one or more target dose areas and one or moreorgans-at-risk areas in the anatomy of the subject, and such image datamay be converted or generated into projections in order for furtherprocessing.

At operation 1920, a trained neural network model is used to generateestimated fluence map representations (fluence maps). For instance, eachof the estimated fluence maps may indicate a fluence distribution of theradiotherapy treatment at a respective beam angle. In a specificexample, each of the estimated fluence maps is a two-dimensional arrayof beamlet weights normal to a respective beam direction, for beamangles of a radiotherapy treatment that correspond to gantry angles of aradiotherapy treatment machine

At operation 1930, the fluence distribution is optimized based on theestimated fluence map representation. For instance, such optimizationmay involve performing numerical optimization with the estimated fluencemaps being provided as input to the optimization, where the optimizationincorporates radiotherapy treatment constraints to produce apareto-optimal fluence plan used in the radiotherapy treatment plan forthe subject.

At operation 1940, a set of initial control points is generated forradiotherapy beams based on the fluence distribution. In an example,this set of initial control points may be generated from performing arcsequencing based on the pareto-optimal fluence plan, to generate a setof initial control points corresponding to each of multiple radiotherapybeams. At operation 1950, a set of final control points is generated forthe radiotherapy beams based on the initial control points. In anexample, this set of final control points may be generated fromperforming direct aperture optimization, to generate a set of finalcontrol points corresponding to each of the multiple radiotherapy beams.

At operation 1960, radiotherapy is delivered with the radiotherapy beamsbased on final control points. In an example, the radiotherapy treatmentis provided as a volume modulated arc therapy (VMAT) radiotherapyperformed by a radiotherapy treatment machine, and the multipleradiotherapy beams are shaped to achieve a modulated coverage of thetarget dose areas from among multiple beam angles, to deliver aprescribed radiation dose. It will be apparent that other methods andoptimizations of radiotherapy treatment may also be used.

FIG. 20 illustrates a block diagram of an example of a machine 2000 onwhich one or more of the methods as discussed herein can be implemented.In one or more examples, one or more items of the image processingdevice 112 can be implemented by the machine 2000. In alternativeexamples, the machine 2000 operates as a standalone device or may beconnected (e.g., networked) to other machines. In one or more examples,the image processing device 112 can include one or more of the items ofthe machine 2000. In a networked deployment, the machine 2000 mayoperate in the capacity of a server or a client machine in server-clientnetwork environment, or as a peer machine in a peer-to-peer (ordistributed) network environment. The machine may be a personal computer(PC), server, a tablet, smartphone, a web appliance, edge computingdevice, a network router, switch or bridge, or any machine capable ofexecuting instructions (sequential or otherwise) that specify actions tobe taken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein.

The example machine 2000 includes processing circuitry or processor 2002(e.g., a CPU, a graphics processing unit (GPU), an ASIC, circuitry, suchas one or more transistors, resistors, capacitors, inductors, diodes,logic gates, multiplexers, buffers, modulators, demodulators, radios(e.g., transmit or receive radios or transceivers), sensors 2021 (e.g.,a transducer that converts one form of energy (e.g., light, heat,electrical, mechanical, or other energy) to another form of energy), orthe like, or a combination thereof), a main memory 2004 and a staticmemory 2006, which communicate with each other via a bus 2008. Themachine 2000 (e.g., computer system) may further include a video displaydevice 2010 (e.g., a liquid crystal display (LCD) or a cathode ray tube(CRT)). The machine 2000 also includes an alphanumeric input device 2012(e.g., a keyboard), a user interface (UI) navigation device 2014 (e.g.,a mouse), a disk drive or mass storage unit 2016, a signal generationdevice 2018 (e.g., a speaker), and a network interface device 2020.

The disk drive unit 2016 includes a machine-readable medium 2022 onwhich is stored one or more sets of instructions and data structures(e.g., software) 2024 embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 2024 mayalso reside, completely or at least partially, within the main memory2004 and/or within the processor 2002 during execution thereof by themachine 2000, the main memory 2004 and the processor 2002 alsoconstituting machine-readable media.

The machine 2000 as illustrated includes an output controller 2028. Theoutput controller 2028 manages data flow to/from the machine 2000. Theoutput controller 2028 is sometimes called a device controller, withsoftware that directly interacts with the output controller 2028 beingcalled a device driver.

While the machine-readable medium 2022 is shown in an example to be asingle medium, the term “machine-readable medium” may include a singlemedium or multiple media (e.g., a centralized or distributed database,and/or associated caches and servers) that store the one or moreinstructions or data structures. The term “machine-readable medium”shall also be taken to include any tangible medium that is capable ofstoring, encoding or carrying instructions for execution by the machineand that cause the machine to perform any one or more of themethodologies of the present disclosure, or that is capable of storing,encoding or carrying data structures utilized by or associated with suchinstructions. The term “machine-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical and magnetic media. Specific examples of machine-readable mediainclude non-volatile memory, including by way of example semiconductormemory devices, e.g., Erasable Programmable Read-Only Memory (EPROM),EEPROM, and flash memory devices; magnetic disks such as internal harddisks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks.

The instructions 2024 may further be transmitted or received over acommunications network 2026 using a transmission medium. Theinstructions 2024 may be transmitted using the network interface device2020 and any one of a number of well-known transfer protocols (e.g.,HTTP). Examples of communication networks include a LAN, a WAN, theInternet, mobile telephone networks, Plain Old Telephone (POTS)networks, and wireless data networks (e.g., Wi-Fi and 4G/5G datanetworks). The term “transmission medium” shall be taken to include anyintangible medium that is capable of storing, encoding or carryinginstructions for execution by the machine, and includes digital oranalog communications signals or other intangible media to facilitatecommunication of such software.

As used herein, “communicatively coupled between” means that theentities on either of the coupling must communicate through an itemtherebetween and that those entities cannot communicate with each otherwithout communicating through the item.

Additional Notes

The above detailed description includes references to the accompanyingdrawings, which form a part of the detailed description. The drawingsshow, by way of illustration but not by way of limitation, specificembodiments in which the disclosure can be practiced. These embodimentsare also referred to herein as “examples.” Such examples can includeelements in addition to those shown or described. However, the presentinventors also contemplate examples in which only those elements shownor described are provided. Moreover, the present inventors alsocontemplate examples using any combination or permutation of thoseelements shown or described (or one or more aspects thereof), eitherwith respect to a particular example (or one or more aspects thereof),or with respect to other examples (or one or more aspects thereof) shownor described herein.

All publications, patents, and patent documents referred to in thisdocument are incorporated by reference herein in their entirety, asthough individually incorporated by reference. In the event ofinconsistent usages between this document and those documents soincorporated by reference, the usage in the incorporated reference(s)should be considered supplementary to that of this document; forirreconcilable inconsistencies, the usage in this document controls.

In this document, the terms “a,” “an,” “the,” and “said” are used whenintroducing elements of aspects of the disclosure or in the embodimentsthereof, as is common in patent documents, to include one or more thanone or more of the elements, independent of any other instances orusages of “at least one” or “one or more.” In this document, the term“or” is used to refer to a nonexclusive or, such that “A or B” includes“A but not B,” “B but not A,” and “A and B,” unless otherwise indicated.

In the appended claims, the terms “including” and “in which” are used asthe plain-English equivalents of the respective terms “comprising” and“wherein.” Also, in the following claims, the terms “comprising,”“including,” and “having” are intended to be open-ended to mean thatthere may be additional elements other than the listed elements, suchthat after such a term (e.g., comprising, including, having) in a claimare still deemed to fall within the scope of that claim. Moreover, inthe following claims, the terms “first,” “second,” and “third,” and soforth, are used merely as labels, and are not intended to imposenumerical requirements on their objects.

Embodiments of the disclosure may be implemented withcomputer-executable instructions. The computer-executable instructions(e.g., software code) may be organized into one or morecomputer-executable components or modules. Aspects of the disclosure maybe implemented with any number and organization of such components ormodules. For example, aspects of the disclosure are not limited to thespecific computer-executable instructions or the specific components ormodules illustrated in the figures and described herein. Otherembodiments of the disclosure may include different computer-executableinstructions or components having more or less functionality thanillustrated and described herein.

Method examples (e.g., operations and functions) described herein can bemachine or computer-implemented at least in part (e.g., implemented assoftware code or instructions). Some examples can include acomputer-readable medium or machine-readable medium encoded withinstructions operable to configure an electronic device to performmethods as described in the above examples. An implementation of suchmethods can include software code, such as microcode, assembly languagecode, a higher-level language code, or the like (e.g., “source code”).Such software code can include computer-readable instructions forperforming various methods (e.g., “object” or “executable code”). Thesoftware code may form portions of computer program products. Softwareimplementations of the embodiments described herein may be provided viaan article of manufacture with the code or instructions stored thereon,or via a method of operating a communication interface to send data viaa communication interface (e.g., wirelessly, over the internet, viasatellite communications, and the like).

Further, the software code may be tangibly stored on one or morevolatile or non-volatile computer-readable storage media duringexecution or at other times. These computer-readable storage media mayinclude any mechanism that stores information in a form accessible by amachine (e.g., computing device, electronic system, and the like), suchas, but are not limited to, floppy disks, hard disks, removable magneticdisks, any form of magnetic disk storage media, CD-ROMS,magnetic-optical disks, removable optical disks (e.g., compact disks anddigital video disks), flash memory devices, magnetic cassettes, memorycards or sticks (e.g., secure digital cards), RAMs (e.g., CMOS RAM andthe like), recordable/non-recordable media (e.g., read only memories(ROMs)), EPROMS, EEPROMS, or any type of media suitable for storingelectronic instructions, and the like. Such computer-readable storagemedium is coupled to a computer system bus to be accessible by theprocessor and other parts of the OIS.

In an embodiment, the computer-readable storage medium may have encodeda data structure for treatment planning, wherein the treatment plan maybe adaptive. The data structure for the computer-readable storage mediummay be at least one of a Digital Imaging and Communications in Medicine(DICOM) format, an extended DICOM format, an XML format, and the like.DICOM is an international communications standard that defines theformat used to transfer medical image-related data between various typesof medical equipment. DICOM RT refers to the communication standardsthat are specific to radiation therapy.

In various embodiments of the disclosure, the method of creating acomponent or module can be implemented in software, hardware, or acombination thereof. The methods provided by various embodiments of thepresent disclosure, for example, can be implemented in software by usingstandard programming languages such as, for example, C, C++, Java,Python, and the like; and combinations thereof. As used herein, theterms “software” and “firmware” are interchangeable, and include anycomputer program stored in memory for execution by a computer.

A communication interface includes any mechanism that interfaces to anyof a hardwired, wireless, optical, and the like, medium to communicateto another device, such as a memory bus interface, a processor businterface, an Internet connection, a disk controller, and the like. Thecommunication interface can be configured by providing configurationparameters and/or sending signals to prepare the communication interfaceto provide a data signal describing the software content. Thecommunication interface can be accessed via one or more commands orsignals sent to the communication interface.

The present disclosure also relates to a system for performing theoperations herein. This system may be specially constructed for therequired purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. The order of execution or performance of the operations inembodiments of the disclosure illustrated and described herein is notessential, unless otherwise specified. That is, the operations may beperformed in any order, unless otherwise specified, and embodiments ofthe disclosure may include additional or fewer operations than thosedisclosed herein. For example, it is contemplated that executing orperforming a particular operation before, contemporaneously with, orafter another operation is within the scope of aspects of thedisclosure.

In view of the above, it will be seen that the several objects of thedisclosure are achieved and other advantageous results attained. Havingdescribed aspects of the disclosure in detail, it will be apparent thatmodifications and variations are possible without departing from thescope of aspects of the disclosure as defined in the appended claims. Asvarious changes could be made in the above constructions, products, andmethods without departing from the scope of aspects of the disclosure,it is intended that all matter contained in the above description andshown in the accompanying drawings shall be interpreted as illustrativeand not in a limiting sense.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with each other. Inaddition, many modifications may be made to adapt a particular situationor material to the teachings of the disclosure without departing fromits scope. While the dimensions, types of materials and coatingsdescribed herein are intended to define the parameters of thedisclosure, they are by no means limiting and are exemplary embodiments.Many other embodiments will be apparent to those of skill in the artupon reviewing the above description. The scope of the disclosureshould, therefore, be determined with reference to the appended claims,along with the full scope of equivalents to which such claims areentitled.

Also, in the above Detailed Description, various features may be groupedtogether to streamline the disclosure. This should not be interpreted asintending that an unclaimed disclosed feature is essential to any claim.Rather, inventive subject matter may lie in less than all features of aparticular disclosed embodiment. Thus, the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate embodiment. The scope of the disclosure should bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled. Further, thelimitations of the following claims are not written inmeans-plus-function format and are not intended to be interpreted basedon 35 U.S.C. § 112, sixth paragraph, unless and until such claimlimitations expressly use the phrase “means for” followed by a statementof function void of further structure.

The Abstract is provided to comply with 37 C.F.R. § 1.72(b), to allowthe reader to quickly ascertain the nature of the technical disclosure.It is submitted with the understanding that it will not be used tointerpret or limit the scope or meaning of the claims.

What is claimed is:
 1. A computer-implemented method for generatingfluence maps used in a radiotherapy treatment plan, the methodcomprising: obtaining a three-dimensional set of image datacorresponding to a subject of radiotherapy treatment, the image dataindicating one or more target dose areas and one or more organs-at-riskareas in the anatomy of the subject; generating anatomy projectionimages from the image data, each anatomy projection image providing aview of the subject from a respective beam angle of the radiotherapytreatment; and using a trained neural network model to generateestimated fluence maps based on the anatomy projection images, each ofthe estimated fluence maps indicating a fluence distribution of theradiotherapy treatment at a respective beam angle, wherein the neuralnetwork model is trained with corresponding pairs of anatomy projectionimages and fluence maps.
 2. The method of claim 1, wherein each of theestimated fluence maps is a two-dimensional array of beamlet weightsnormal to a respective beam direction, and wherein beam angles of theradiotherapy treatment correspond to gantry angles of a radiotherapytreatment machine.
 3. The method of claim 2, wherein obtaining thethree-dimensional set of image data corresponding to a subject includesobtaining image data for each gantry angle of the radiotherapy treatmentmachine, and wherein each generated anatomy projection image representsa view of the anatomy of the subject from a given gantry angle used toprovide treatment with a given radiotherapy beam.
 4. The method of claim1, further comprising: using the estimated fluence maps to determineradiation doses in the radiotherapy treatment plan, wherein theradiotherapy treatment comprises a volume modulated arc therapy (VMAT)radiotherapy performed by a radiotherapy treatment machine, whereinmultiple radiotherapy beams are shaped to achieve a modulated dose fortarget areas, from among multiple beam angles, to deliver a prescribedradiation dose.
 5. The method of claim 1, wherein each anatomyprojection image is generated by forward projection of thethree-dimensional set of image data from respective angles of multiplebeam angles.
 6. The method of claim 1, wherein training the neuralnetwork model uses pairs of anatomy projection images and fluence mapsfrom a plurality of human subjects, wherein each individual pair isprovided from a same human subject, and wherein the neural network modelis trained with operations comprising: obtaining multiple sets oftraining anatomy projection images, each set of the training anatomyprojection images indicating one or more target dose areas and one ormore organs-at-risk areas in the anatomy of a respective subject;obtaining multiple sets of training fluence maps corresponding to thetraining anatomy projection images, each set of the training fluencemaps indicating a fluence distribution for the respective subject; andtraining the neural network model based on the training anatomyprojection images that correspond to the training fluence maps.
 7. Themethod of claim 6, wherein the corresponding pairs of the anatomyprojection images and the fluence maps used for training the neuralnetwork model are obtained for each beam angle of a radiotherapytreatment machine.
 8. The method of claim 6, wherein the neural networkmodel is a generative model of a generative adversarial network (GAN)comprising at least one generative model and at least one discriminativemodel, wherein the at least one generative model and the at least onediscriminative model correspond to respective generative anddiscriminative convolutional neural networks.
 9. The method of claim 8,wherein the GAN comprises a conditional adversarial network (cGAN) or acycle-consistent generative adversarial network (CycleGAN).
 10. Themethod of claim 8, wherein the GAN is configured to train the generativemodel using a discriminative model, wherein neural network parametervalues learned by the generative model and the discriminative model areestablished using adversarial training between the discriminative modeland the generative model, and wherein the adversarial trainingcomprises: training the generative model to generate a first estimatedfluence map at a first beam angle from a projection image thatrepresents a view of a training subject anatomy from the first beamangle; and training the discriminative model to classify the firstestimated fluence map as an estimated or as a real training fluence mapprojection image; and wherein an output of the generative model is usedfor training the discriminative model and an output of thediscriminative model is used for training the generative model.
 11. Themethod of claim 8, wherein the GAN is a cycle-consistent generativeadversarial network (CycleGAN) comprising the generative model and thediscriminative model, wherein the generative model is a first generativemodel and the discriminative model is a first discriminative model,wherein the CycleGAN further comprises: a second generative modeltrained to: process, from a given pair of the pairs of anatomyprojection images and fluence maps, a given fluence map at a given beamangle as an input; and generate an estimated anatomy projection imagethat represents a view of a subject anatomy from the given beam angle asan output; and a second discriminative model trained to classify theestimated anatomy projection image as an estimated or as a real anatomyprojection image.
 12. The method of claim 11, wherein the CycleGANcomprises a first portion to train the first generative model, the firstportion being trained to: obtain a set of training anatomy projectionimages representing different views of a patient anatomy from priortreatments that are paired with a set of training fluence mapscorresponding to each of the different views, each of the trainingfluence maps being aligned with a respective one of the training anatomyprojection images; input the set of training anatomy projection imagesto the first generative model, and output a set of estimated fluencemaps from the first generative model; input the set of estimated fluencemaps to the first discriminative model, and classify the set ofestimated fluence maps with the first discriminative model as anestimated or as a real set of fluence maps; and input the set of fluencemaps to the second generative model, and generate a set of estimatedanatomy projection images, to calculate the cycle-consistency losses.13. The method of claim 12, wherein the CycleGAN comprises a secondportion that is trained to: input the set of training fluence mapscorresponding to each of the different views to the second generativemodel, and output a set of generated anatomy projection images from thesecond generative model; input the set of generated anatomy projectionimages to the second discriminative model, and classify the set ofgenerated anatomy projection images as an estimated or real set ofanatomy projection images; and input the set of anatomy projectionimages to the first generative model to generate a set of estimatedfluence maps to calculate the cycle-consistency losses.
 14. The methodof claim 12, wherein: the cycle-consistency losses are generated basedon: a) a comparison of the set of generated anatomy projection imageswith the set of training anatomy projection images, and b) a comparisonof the set of generated fluence maps with the set of training fluencemaps; wherein the first generative model is trained to minimize a firstloss term that represents an expectation of a difference between aplurality of estimated fluence map representation images andrespectively paired training fluence maps; and wherein the secondgenerative model is trained to minimize a second loss term thatrepresents an expectation of difference between a plurality of estimatedanatomy projection images and respectively paired training anatomyprojection images.
 15. The method of claim 1, further comprising:generating a set of estimated fluence maps using the neural networkmodel; and performing numerical optimization with the estimated fluencemaps as input to the optimization, where the optimization incorporatesradiotherapy treatment constraints, to produce a pareto-optimal fluenceplan used in the radiotherapy treatment plan for the subject.
 16. Themethod of claim 15, further comprising: performing arc sequencing basedon the pareto-optimal fluence plan, to generate a set of initial controlpoints corresponding to each of multiple radiotherapy beams; andperforming direct aperture optimization, to generate a set of finalcontrol points corresponding to each of the multiple radiotherapy beams.17. The method of claim 16, wherein the radiotherapy treatment comprisesa volume modulated arc therapy (VMAT) radiotherapy performed by aradiotherapy treatment machine, and wherein the arc sequencing based onthe pareto-optimal fluence plan is performed such that the multipleradiotherapy beams are shaped to achieve a modulated coverage of thetarget dose areas, from among multiple beam angles, to deliver aprescribed radiation dose.
 18. The method of claim 16, furthercomprising: performing a radiotherapy treatment, using the set of finalcontrol points, wherein the set of final control points are used tocontrol multi-leaf collimator (MLC) leaf positions of a radiotherapytreatment machine at a given gantry angle corresponding to a given beamangle.
 19. The method of claim 1, further comprising: comparing afluence map produced from the neural network model in response to aninput set of anatomy projection images, with a fluence map produced fromanother source.
 20. A system for generating fluence maps used in aradiotherapy treatment plan, the system comprising: one or more memorydevices to store a three-dimensional set of image data corresponding toa subject of radiotherapy treatment, the image data indicating one ormore target dose areas and one or more organs-at-risk areas in theanatomy of the subject; and one or more processors configured to performoperations comprising: obtaining anatomy projection images from theimage data, each anatomy projection image providing a view of thesubject from a respective beam angle of the radiotherapy treatment; andexecuting a trained neural network model to generate computer-estimatedfluence maps based on input of the anatomy projection images, each ofthe estimated fluence maps indicating a fluence distribution of theradiotherapy treatment at a respective beam angle, wherein the neuralnetwork model is trained with corresponding pairs of anatomy projectionimages and fluence maps.
 21. The system of claim 20, the one or moreprocessors further configured to perform operations comprising:generating a set of estimated fluence maps using the neural networkmodel; and performing numerical optimization with the estimated fluencemaps as input to the optimization, where the optimization incorporatesradiotherapy treatment constraints, to produce a pareto-optimal fluenceplan used in the radiotherapy treatment plan for the subject; whereineach of the estimated fluence maps is a two-dimensional array of beamletweights normal to a respective beam direction, and wherein beam anglesof the radiotherapy treatment correspond to gantry angles of aradiotherapy treatment machine.
 22. The system of claim 21, the one ormore processors further configured to perform operations comprising:performing arc sequencing based on the pareto-optimized fluence plan, togenerate a set of initial control points corresponding to each ofmultiple radiotherapy beams; and performing direct apertureoptimization, to generate a set of final control points corresponding toeach of the multiple radiotherapy beams.
 23. The system of claim 20, theone or more processors further configured to perform operationscomprising: using the estimated fluence maps to determine radiationdoses in the radiotherapy treatment plan, wherein the radiotherapytreatment comprises a volume modulated arc therapy (VMAT) radiotherapyperformed by a radiotherapy treatment machine, wherein multipleradiotherapy beams are shaped to achieve a modulated dose for targetareas, from among multiple beam angles, to deliver a prescribedradiation dose.
 24. The system of claim 23, wherein training the neuralnetwork model uses pairs of anatomy projection images and fluence mapsfrom a plurality of human subjects, wherein each individual pair isprovided from a same human subject, and wherein the neural network modelis trained with operations comprising: obtaining multiple sets oftraining anatomy projection images, each set of the training anatomyprojection images indicating one or more target dose areas and one ormore organs-at-risk areas in the anatomy of a respective subject;obtaining multiple sets of training fluence maps corresponding to thetraining anatomy projection images, each set of the training fluencemaps indicating a fluence distribution for the respective subject; andtraining the neural network model based on the training anatomyprojection images that correspond to the training fluence maps; whereinthe neural network model is a generative model of a generativeadversarial network (GAN) comprising at least one generative model andat least one discriminative model, wherein the at least one generativemodel and the at least one discriminative model correspond to respectivegenerative and discriminative convolutional neural networks.
 25. Anon-transitory computer-readable storage medium comprisingcomputer-readable instructions for generating fluence maps used in aradiotherapy treatment plan, the instructions performing operationscomprising: identifying a three-dimensional set of image datacorresponding to a subject of radiotherapy treatment, the image dataindicating one or more target dose areas and one or more organs-at-riskareas in the anatomy of the subject; generating anatomy projectionimages from the image data, each anatomy projection image providing aview of the subject from a respective beam angle of the radiotherapytreatment; and using a trained neural network model to generatecomputer-estimated fluence maps based on input of the anatomy projectionimages, each of the estimated fluence maps indicating a fluencedistribution of the radiotherapy treatment a respective beam angle,wherein the neural network model is trained with corresponding pairs ofanatomy projection training images and fluence maps.
 26. Thecomputer-readable storage medium of claim 25, the instructions furtherperforming operations comprising: generating a set of estimated fluencemaps using the neural network model; and performing numericaloptimization with the estimated fluence maps as input to theoptimization, where the optimization incorporates radiotherapy treatmentconstraints, to produce a pareto-optimal fluence plan used in theradiotherapy treatment plan for the subject; wherein each of theestimated fluence maps is a two-dimensional array of beamlet weightsnormal to a respective beam direction, and wherein beam angles of theradiotherapy treatment correspond to gantry angles of a radiotherapytreatment machine.
 27. The computer-readable storage medium of claim 26,the instructions further performing operations comprising: performingarc sequencing based on the pareto-optimal fluence plan, to generate aset of initial control points corresponding to each of multipleradiotherapy beams; and performing direct aperture optimization, togenerate a set of final control points corresponding to each of themultiple radiotherapy beams.
 28. The computer-readable storage medium ofclaim 25, the instructions further performing operations comprising:using the estimated fluence maps to determine radiation doses in theradiotherapy treatment plan, wherein the radiotherapy treatmentcomprises a volume modulated arc therapy (VMAT) radiotherapy performedby a radiotherapy treatment machine, wherein multiple radiotherapy beamsare shaped to achieve a modulated dose for target areas, from amongmultiple beam angles, to deliver a prescribed radiation dose.
 29. Thecomputer-readable storage medium of claim 25, wherein training theneural network model uses pairs of anatomy projection images and fluencemaps from a plurality of human subjects, wherein each individual pair isprovided from a same human subject, and wherein the neural network modelis trained with operations comprising: obtaining multiple sets oftraining anatomy projection images, each set of the training anatomyprojection images indicating one or more target dose areas and one ormore organs-at-risk areas in the anatomy of a respective subject;obtaining multiple sets of training fluence maps corresponding to thetraining anatomy projection images, each set of the training fluencemaps indicating a fluence distribution for the respective subject; andtraining the neural network model based on the training anatomyprojection images that correspond to the training fluence maps; whereinthe neural network model is a generative model of a generativeadversarial network (GAN) comprising at least one generative model andat least one discriminative model, wherein the at least one generativemodel and the at least one discriminative model correspond to respectivegenerative and discriminative convolutional neural networks.